Hi Eric,
comments inside...
Am 24.05.2011 06:08, schrieb Eric Charles:
Hi,
For the immutable mails:
1. if we store each mail in a file, we don't have the alter it but we
face the performance issue cause reading a small file in Hadoop seems
expensive (not performant).
Seems like this, yeah..
2. If we store each folder in a file, we may have less performance
issue on read (larger file), but we face the issue that we can not
alter the content (only append!!). So does not sound like an option.
Well we could just have some kind of info which mails are deleted and
skip then while read from the file. This would still need to cleanup
"deleted" messages later somehow. Not sure if it makes sense
given by the complexibilty it will introduce..
For associated metadata, maildir offers this functionality by using
the file name as metadata container. On change, the file is renamed
addign some flags,... which is possible with Hadoop ([4] for example
operations on hdfs). Once again, at the price of performance for small
size.
As Robert suggested in [1], a benchmark could be setup, but we would
need a realistic cluster (numerous hardware machines with replication
factor >= 3) and large dataset (millions of mails) to get some
representative numbers.
On the possible file format, we have a limited options (hadoop calls
these some Writable): Text or BytesWritable. There's also file-based
data structures: SequenceFile or MapFile.
I also answered on [1] asking what hadoop can offer in regards to Avro
format (see also [5] on the protocol buffer, avro kind-of, usage at
twitter). I don't know if Avro file format changes anything to the
exposed considerations...
In this Hadoop approach, we also need to ask how we get/query the
information. Directly read the Hadoop Writable/File via the io API, or
use a map/reduce job ? The map/reduce job result will be stored in a
OutputFile which must in its turn be read again, sounds a bit too much
to me...
Now, if we find all these too challenging and we are not sure we will
get a performant solution, HBase for example is a proven solution and
offers a structured storage on top of Hadoop.
There's some ORM around (like the datanucleus jdo,...) but the HBase
native API is rich enough and should do the job for us without
additional layer.
+1, for no ORM ;)
I am following the Apache Gora incubating mailing list as it seems to
have much to offer (persistence towards hbase, cassandra,..
indexing...) but the last time the project seemed to be quiet. This
doesn't mean the today functionality is not usable for us.
Another question is about the potential usage of the existing lucene
index to help us on the queries (for IMAP, currently in mailbox-store
project). This would be a nice solution to use, but today the index is
local (not distributed). It's a work in progress, and can evolve
towards distribution. I don't think we need to decide on this now, but
the question will come one day.
Unfortunally the Lucene Index is not complete yet, its still on my todo
list ;)
Tks,
- Eric
[4]
http://myjavanotebook.blogspot.com/2008/05/hadoop-file-system-tutorial.html
[5]
http://www.slideshare.net/kevinweil/protocol-buffers-and-hadoop-at-twitter
On 24/05/2011 00:01, Ioan Eugen Stan wrote:
Hello,
I had some discussions with Eric about what will be the best way to
implement the mailbox over HDFS and we agreed that it's better to
inform the list about the situation.
The project idea that I applied for is to implement James mailbox
storage over Hadoop HDFS and one of the first steps was to find the
best way to interact with Hadoop. So I just did that. I have spent the
last week or so trying to figure out the best way to implement the
mailbox over Hadoop. I found the training videos from Cloudera to be
very helpful [2].
I also wrote on the Hadoop mailing list to ask them for an opinion
(before watching the videos) . You can read the discussion here [1].
I have come to the conclusion that there is no easy way to implement
the mailbox directly over HDFS, and my opinion is to use HBase, either
directly or over Gora. I will support my statement with some of the
things I found out.
First, about email :
- emails are essentially immutable. Once created they do not modify.
- meta information is read/write (like the status - read/unread).
maybe other stuff, I still have to get up to date.
- you can delete an email, but other than that you can't modify it.
- users usually access the last 50-100 emails (my observation)
About HDFS:
- is designed to work well with large data with the order of magnitude
of GB and beyond. It has a block size> 64 MB. This enables less disk
seeks when reading a file, because the file is less fragmented. It
uses bulk reads and writes enables to HDFS to perform better: all the
data is in one place, and you have a small number of open file
handlers, which means less over-heed.
- does not provide random file alteration. HDFS only supports APPEND
information at the end of an existing file. If you need to modify a
file, the only way to do it is to create a new file with the
modifications.
HBase:
- is a NoSQL implementation over Hadoop.
- provides the user a way to store information and access it very
easily based on some keys.
- provides a way to modify the files by keeping a log, similar to the
way journal file systems work: it appends all the modifications to a
log file. When certain conditions are met the log file is merged back
into the „database”.
My conclusions:
Because emails are small and require that a part of them needs to be
changed, storing them in a filesystem that was designed for large
files, which does not provide a way to modify these files is not a
sensible thing to do.
I see a couple of choices:
1. we use HBase
2. we keep the meta information in a separate database, outside
Hadoop, but things will not scale very well.
3. we design it on top of HDFS, but essentially we (I) will end up
solving the same problems that HBase solved
The most easy and straight forward solution is to use HBase, There is
a paper [3] that shows some results with an email store based on
Cassandra, so it is proven to work.
I am thinking of using Gora and avoiding to use HBase API directly.
This will ensure that James could use any NoSQL storage that Gora can
access. What keeps me back is that Gora does not seem to be very
active and it's also incubating so I may run into things not easy to
get out of.
What do you think?
[1]
http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.user/26022
[2] http://vimeo.com/search/videos/search:cloudera/st/48b36a32
[3] http://ewh.ieee.org/r6/scv/computer/nfic/2009/IBM-Jun-Rao.pdf
Bye,
Norman
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
