[CONF] Apache Qpid > Persistent Cluster Restart Design Note

[confluence]

Persistent Cluster Restart Design Note Page edited by Alan Conway Persistent clusters. A cluster has either all transient or all persistent members, mixed clusters are not allowed. Persistent restart scenarios: first run of persistent cluster, all members have empty stores. persistent member crashes is re started - re-joins running cluster automatic restart after orderly shutdown of persistent cluster manual recovery after total cluster failure of persistent cluster Other requirements: cluster initialization: wait for N initial members before going active. enforce consistency of broker options that need to be identical across cluster Wait for N New option: cluster-wait-for N. Wait for at least N initial members before going active. guarantees that clients will not be served till N members are active. check consistency of all N members (options, store state) before starting. members with clean store can restore from store, don't need an update. Can be used with transient and persistent clusters. Transient cluster Initialization Wait for N initial members Verify options are consistent for all members or abort. All members become active. Joining Check options are consistent with cluster. Get update from an active broker, become active. Persistent cluster Store statess on broker start-up: empty: not used before. clean: has state, was shut down by admin. Has intial and final UUIDs dirty: has state, not shut down by admin. Has initial UUID. Initial UUID is stored on the first run of a persistent cluster. Used to ensure members are part of the same cluster. Final UUID is stored at administrative shut-down of the cluster. Used to ensure clean stores are from the same shut-down event. Initialization Wait for N initial members Verify options are consistent for all members or abort. Verify valid store states or abort (see below) Members with empty/dirty stores get update from clean member. All empty is a valid store state: all members record the same initial UUID and go active. If any are non empty then at least one store must be clean all clean stores must have same final UUID. all clean and dirty stores must have same initial UUID. All clean members restore from stores. All empty members set the initial UUID from the cluster. All dirty/empty members get an update from a clean member. Joining If the new member has a non-empty store, the initial UUID must match the cluster. The new member gets an update from the cluster. Manual Recovery If the entire cluster fails then manual recovery is required. While running brokers will peridiocally (on every membership change and at some configured time interval) write a sequence number to disk. Provide tools to examine broker data directories and determine if they belong to the same cluster (same initial UUID) and if so which is the latest based on the sequence number. Recovery procedure is to mark the latest store as clean and restart the cluster. Change Notification Preferences View Online | View Change | Add Comment --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:commits-subscr...@qpid.apache.org