Jules,
This is awesome stuff. I look forward to playing with this stuff in
Geronimo. Let me know if you need a hand with something...
Jeff
Jules Gosnell wrote:
Guys,
I thought that it was high time that I brought you up to date with my
efforts in building a clustering layer for Geronimo.
The project, wadi.codehaus.org, started as an effort to build a
scalable clustered HttpSession implementation, but in doing this, I
have built components that should be useful in clustering the state
held in any tier of Geronimo e.g. OpenEJB SFSBs etc.
WADI (Web Application Distribution Infrastructure) has two main
elements - the vertical and the horizontal.
Vertically, WADI comprises a stack of pluggable stores. Each store has
a pluggable Evicter responsible for demoting aging Sessions
downwards. Requests arriving at the container are fed into the top of
the stack and progress downwards, until their corresponding Session is
found and promoted to the top, where the request is correctly rendered
in its presence.
Typically the top-level store is in Memory. Aging Sessions are demoted
downwards onto exclusively owned LocalDisc. The bottom-most store is a
database shared between all nodes in the Cluster. The first node
joining the Cluster promotes all Sessions from the database into
exclusively-owned store - e.g. LocalDisc. The last node to leave the
Cluster demotes all Sessions down back into the database.
Horizontally, all nodes in a WADI Cluster are connected (p2p) via a
Clustered Store component within this stack. This typically sits at
the boundary between exclusive and shared Stores. As requests fall
through the stack, looking for their corresponding Session they arrive
at the Clustered store, where, if the Session is present anywhere in
the Cluster, its location may be learnt. At this point, the Session
may be migrated in, underneath the incoming request, or, if its
current location is considered advantageous, the request may be
proxied or redirected to its remote location. As a node leaves the
Cluster, all its Sessions are evacuated to other nodes via this store,
so that they may continue to be actively maintained.
The space in which Session ids are allocated is divided into a fixed
number of Buckets. This number should be large enough such that
management of the Buckets may be divided between all nodes in the
Cluster roughly evenly. As nodes leave and join the Cluster, a single
node, the Coordinator, is responsible for re-Bucketing the Cluster -
i.e. reorganising who manages which Buckets and ensuring the safe
transfer of the minimum number of Buckets to implement the new
layout. The Coordinator is elected via a Pluggable policy. If the
Coordinator leaves or fails, a new one is elected. If a node leaves or
joins, buckets emigrate from it or immigrate into it, under the
control of the Coordinator, to/from the rest of the Cluster.
A Session may be efficiently mapped to a Bucket by simply %-ing its
ID's hashcode() by the number of Buckets in the Cluster.
A Bucket is a map of SessionID:Location, kept up to date with the
Location of every Session in the Cluster, of which the id falls into
its space. i.e. as Sessions are created, destroyed or moved around the
Cluster notifications are sent to the node managing the relevant
Bucket, informing it of the change.
In this way, if a node receives a request for a Session which it does
not own locally, it may pass a message to it, in a maximum of
typically two hops, by sending the message to the Bucket owner, who
then does a local lookup of the Sessions actual location and forwards
the message to it. If Session and Bucket can be colocated, this can
reduced to a single hop.
Thus, WADI provides a fixed and scalable substrate over the more fluid
arrangement that Cluster membership comprises, on top of which further
Clustered services may be built.
The above functionality exists in WADI CVS and I am currently working
on hardening it to the point that I would consider it production
strength. I will then consider the addition of some form of state
replication, so that, even with the catastrophic failure of a member
node, no state is lost from the Cluster.
I plan to begin integrating WADI with Geronimo as soon as a certified
1.0-based release starts to settle down. Certification is the most
immediate goal and clustering is not really part of the spec, so I
think it best to satisfy the first before starting on subsequent
goals.
Further down the road we need to consider the unification of session
id spaces used by e.g. the web and ejb tiers and introduction of an
ApplicationSession abstraction - an object which encapsulates all
e.g. web and ejb sessions associated with a particular client talking
to a particular application. This will allow WADI to maintain the
colocation of associated state, whilst moving and replicating it
around the Cluster.
If anyone would like to know more about WADI, please feel free to ask
me questions here on geronimo-dev or on wadi-dev.
Thanks for listening,
Jules
