Also, on replication/distribution, we have two distinct aspects:
1. *map replication* - the pointers map has to be replicated to all nodes
and each pointer have to contain also a reference to the node who "owns" the
real data
2. *communication between nodes* - once one node knows that one entry is
contained in node "n" has to ask for it
The first point is easily covered by terracotta or hazelcast, while the
second one should be implemented using an RPC mechanism (Thrift or Avro are
both good choices). Another option is to cover also point 1 with a custom
replication built on top of the chosen RPC framework - of course this would
lead to another (do we really need it?) distributed map implementation.
Keeping things like this is easy - of course making it efficient/performant
is a different story (i.e., should I keep a local cache of frequently
accessed items stored in other nodes? etc..).
Ciao,
R
On Wed, Oct 19, 2011 at 8:43 AM, Raffaele P. Guidi <
[email protected]> wrote:
> I Used to have two file storage implementations: the simplest one which
> wrote every entry on disk as a separate file - terribly slow - and the
> second one based on OrientDB, a nosql solution - fast but used all the heap
> on its own - you can see my blog for reference. Maybe the JCS file storage
> could fit?
>
> On Wednesday, October 19, 2011, Ashish <[email protected]> wrote:
> > On Wed, Oct 19, 2011 at 11:02 AM, Akash Ashok <[email protected]>
> wrote:
> >> Thanks Guys for responding on a quick reply. HBase is indeed my first
> love
> >> :)
> >>
> >> I was really interested in what DirectMemory could offer and am excited
> to
> >> see that its pretty awesome. I posed that question because I wasn't
> really
> >> sure as to what the rules were on the incubator :) Great to know I
> could
> >> get started right away. Shall start digging thru the code.
> >
> > All ASF projects work the same way :)
> >
> >>
> >> I went through the road map and have a few questions:
> >>
> >> 1. File Storage -
> >> This was my main concern. This moves more towards a database approach or
> to
> >> be more specific Key Value store approach Aren't there existing file
> based
> >> solutions which can be used instead of developing another one?
> >
> > Yes there are. We can use any of key-value stores or Object DB's like
> > BerkleyDB or others.
> > It would be good to have a pluggable layer, where a persistant
> > provider can be chosen.
> > However, its OffHeap aka stored in RAM, so don't feel its a priority,
> > and persistance would make sense when we evolve into a Cache
> > framework.
> >
> >>
> >> 2. Hazelcast for replication -
> >> If i get this right the idea to use it more like a replication framework
> and
> >> then use that to cache them onto the DirectMemory on the respective
> systems
> >> ?
> >
> > Hazelcast already has released OffHeap storage. Moreover, its a
> > complete Cache framework with provision of adding persistant stores.
> > So I am very much against using it. Please note that I use Hazelcast
> > myself, so not implying that its not a good framework.
> >
> > Going Distributed is a different story, and if you see the past
> > conversations, our current focus is on OffHeap store.
> > To go distributed, we have to move scope from OffHeap store to a Cache
> > solution. Then comes the challenges Distributed or Replicated.
> > Replicated is easy, but Distributed world would bring in a lot more
> > design choice like Server or peer2peer architecture, what concurrency
> > levels to support etc.
> >
> > My take
> > 1. Implement OffHeap store
> > 2. Benchmark it, refine it to give latencies that we need in
> > production (would luv sub-milliseconds or keep then less than 2 ms)
> >
> > Once we reach here, we can think about going complete caching
> > solution, followed by going distributed :)
> >
> >>
> >> Lastly "I recently rewrote DM entirely for simplification." +1 on the
> fact
> >> of simplification.
> >> As Einstien once said " Make things as simple as possible, not simpler"
> :)
> >>
> >> Cheers,
> >> Akash A
> >>
> >>
> >>
> >> On Wed, Oct 19, 2011 at 8:33 AM, Raffaele P. Guidi <
> >> [email protected]> wrote:
> >>
> >>> Uhm, I'm new in the ASF, but I guess you just need to read some docs
> (ok,
> >>> we
> >>> don't have many, just something in the old wiki @github and an initial
> >>> roadmap proposal [1]), take a look at the code and the issues and get
> >>> started :) I see googling around that you have interests in hadoop and
> >>> hbase
> >>> as well, so maybe you could start investigating sinergies with those
> >>> products (just an idea, it is in the roadmap).
> >>>
> >>> In any case thanks for your interest and welcome aboard.
> >>>
> >>> Ciao,
> >>> R
> >>>
> >>> [1] Roadmap proposal:
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/DIRECTMEMORY/2011/10/18/Apache+DirectMemory+-+initial+roadmap+discussion
> >>>
> >>> On Wed, Oct 19, 2011 at 4:44 AM, Akash Ashok <[email protected]>
> >>> wrote:
> >>>
> >>> > I know this is still getting incubated. But I would like to get
> involved
> >>> in
> >>> > the development. What are the rules like? Can I get involved ?
> >>> >
> >>> > Cheers,
> >>> > Akash A
> >>> >
> >>>
> >>
> >
> >
> >
> > --
> > thanks
> > ashish
> >
> > Blog: http://www.ashishpaliwal.com/blog
> > My Photo Galleries: http://www.pbase.com/ashishpaliwal
> >
>
