The branch, 1.2.40 has been updated via 8b2d84482bacd3b31db013496ce82c2e7b730e86 (commit) via e78e37205308e1507a2cf86a655a95893a7cd413 (commit) via cfb85046e6ed87bf01c3abe8cc908a6d2be741e1 (commit) via 5c44156d1aea799f1d6655dd0237e01c49027b82 (commit) via bde1c733fc8c4009202bf185452914f17631c1e7 (commit) via 6479566a0a104b903f499979db594541ffc00a1f (commit) via 5205d545e8d8c72d73b9d5fd148df6de30392fc8 (commit) from b7467294465b6225982c90315df20a8699ccf812 (commit)
http://gitweb.samba.org/?p=ctdb.git;a=shortlog;h=1.2.40 - Log ----------------------------------------------------------------- commit 8b2d84482bacd3b31db013496ce82c2e7b730e86 Author: Amitay Isaacs <ami...@gmail.com> Date: Tue Nov 27 15:50:54 2012 +1100 New version 1.2.55 Signed-off-by: Amitay Isaacs <ami...@gmail.com> commit e78e37205308e1507a2cf86a655a95893a7cd413 Author: Amitay Isaacs <ami...@gmail.com> Date: Thu Nov 22 14:37:45 2012 +1100 Revert "when creating/adding a public ip, set the initial interface to be the first interface specified" This reverts commit 4308935ba48ac7a29e7523315acf580019715f0f. When IP is added to a node on a new interface for the first time, vnn->iface gets set to the first interface defined for that IP. This actually causes problem in ctdb_vnn_assign_iface(). Since vnn->iface is set it takes an early exit without updating vnn->pnn. This results in IP being hosted on the node, but CTDB still thinks it's unassigned. Signed-off-by: Amitay Isaacs <ami...@gmail.com> commit cfb85046e6ed87bf01c3abe8cc908a6d2be741e1 Author: Martin Schwenke <mar...@meltin.net> Date: Fri Nov 16 20:21:15 2012 +1100 Eventscripts: 10.interface should list configured interfaces The current code lists available interfaces. If IPs are configured in some other way than the public addresses file (e.g. ctdb addip) and their interfaces default to being marked down then, since down interfaces are not available, these interfaces can never be marked up. The configured interfaces should be listed instead. Signed-off-by: Martin Schwenke <mar...@meltin.net> Cherry-pick-from: d8f010355b715e49709836e057a5d0f110919897 Conflicts: config/events.d/10.interface commit 5c44156d1aea799f1d6655dd0237e01c49027b82 Author: Martin Schwenke <mar...@meltin.net> Date: Fri Nov 16 19:43:14 2012 +1100 ctdbd: Make the link status of new interfaces more flexible Neither up nor down is a good default value for the link status of a new interface. Up means that IPs can be assigned to interfaces before the true state is known and they can move away quickly if the interface is actually down. Down means that IPs can't be assigned to an interface for a variable amount of time - until a monitor cycle occurs - and this can result in imbalanced IPs. This is a neat compromise. Before the startup event completes, IPs can't be assigned to interfaces because all interfaces begin in a down state. As soon as the startup event completes, IPs can be allocated to any interface that has been marked up by the eventscript. Later, during normal operation, newly added IPs can be assigned to new interfaces immediately. The IPs will still move away if an interface is noticed to be down in the next monitor cycle, but that is the exception rather than the rule. Signed-off-by: Martin Schwenke <mar...@meltin.net> Cherry-pick-from: 9275a69a414482f1053ae14528d5972575b9214e commit bde1c733fc8c4009202bf185452914f17631c1e7 Author: Amitay Isaacs <ami...@gmail.com> Date: Tue Nov 6 17:06:54 2012 +1100 tools/ctdb: Do not use function return value as pnn This fixes the wrong code where same variable 'ret' is used to track the pnn and the return value of a function call. Signed-off-by: Amitay Isaacs <ami...@gmail.com> Cherry-pick-from: 718233c445cd6627ab3962b6565c2655f1f8efd0 commit 6479566a0a104b903f499979db594541ffc00a1f Author: Amitay Isaacs <ami...@gmail.com> Date: Tue Oct 23 16:23:12 2012 +1100 recoverd: Track the nodes that fail takeover run and set culprit count If any of the nodes fail takeover run (either due to timeout or failure to complete within takeover_timeout interval) from main loop, recovery master will give up trying takeover run with following message: "Unable to setup public takeover addresses. Try again later" And as a side-effect the monitoring is disabled on all the nodes. Before ctdb_takeover_run() is called from main loop, monitoring get disabled via startrecovery event. Since ctdb_takeover_run() fails, it never runs recovered event and monitoring does not get re-enabled. In main_loop, ctdb_takeover_run() is called with a takeover_fail_callback. This callback will get called if any of the nodes fail in handling takeip/releaseip/ipreallocated events in ctdb_takeover_run(). Signed-off-by: Amitay Isaacs <ami...@gmail.com> Cherry-pick-from: a5c6bb1fffb8dc3960af113957a1fd080cc7c245 Conflicts: include/ctdb_private.h server/ctdb_takeover.c commit 5205d545e8d8c72d73b9d5fd148df6de30392fc8 Author: Amitay Isaacs <ami...@gmail.com> Date: Tue Oct 23 15:06:33 2012 +1100 daemon: Do not ignore timed out monitor events If an eventscript times out for monitor event, it is considered successful and the remaining eventscripts are not run. This can make a node prematurely healthy, cause healthy node to fail over IPs to this node and this node will not be able to host those IPs. Thus causing loss of access and in case of NAT-GW configuration, loss of a default route. Copy-code-from: 6e68797af67bee36f2bad045f94806e7e98f27e9 Signed-off-by: Amitay Isaacs <ami...@gmail.com> ----------------------------------------------------------------------- Summary of changes: config/events.d/10.interface | 6 +++--- include/ctdb_private.h | 6 +++--- packaging/RPM/ctdb.spec.in | 8 +++++++- server/ctdb_recoverd.c | 34 +++++++++++++++++++++++++++++----- server/ctdb_takeover.c | 32 +++++++++++++++++++++++++------- server/ctdb_tunables.c | 2 +- server/eventscript.c | 1 - tools/ctdb.c | 8 +++++--- 8 files changed, 73 insertions(+), 24 deletions(-) Changeset truncated at 500 lines: diff --git a/config/events.d/10.interface b/config/events.d/10.interface index 8f0c0bc..d407154 100755 --- a/config/events.d/10.interface +++ b/config/events.d/10.interface @@ -25,9 +25,9 @@ monitor_interfaces() [ "$CTDB_NATGW_PUBLIC_IFACE" ] && INTERFACES="$CTDB_NATGW_PUBLIC_IFACE $INTERFACES" - # For all but the 1st line, get the 2nd last field with commas - # changes to spaces. - local IFACES=`ctdb -Y ip -v | sed -e '1d' -e 's/:[^:]*:$//' -e 's/^.*://' -e 's/,/ /g'` + # Get the configured interfaces for each IP. That is, for all but + # the 1st line, get the last field with commas changed to spaces. + local IFACES=$(ctdb -Y ip -v | sed -e '1d' -e 's/:$//' -e 's/^.*://' -e 's/,/ /g') local IFACE diff --git a/include/ctdb_private.h b/include/ctdb_private.h index c54e1b0..3ff5957 100644 --- a/include/ctdb_private.h +++ b/include/ctdb_private.h @@ -1132,6 +1132,8 @@ int ctdb_sys_send_tcp(const ctdb_sock_addr *dest, const ctdb_sock_addr *src, uint32_t seq, uint32_t ack, int rst); +typedef void (*client_async_callback)(struct ctdb_context *ctdb, uint32_t node_pnn, int32_t res, TDB_DATA outdata, void *callback_data); + int ctdb_set_public_addresses(struct ctdb_context *ctdb, const char *alist); int ctdb_set_single_public_ip(struct ctdb_context *ctdb, const char *iface, @@ -1139,7 +1141,7 @@ int ctdb_set_single_public_ip(struct ctdb_context *ctdb, int ctdb_set_event_script(struct ctdb_context *ctdb, const char *script); int ctdb_set_event_script_dir(struct ctdb_context *ctdb, const char *script_dir); int ctdb_set_notification_script(struct ctdb_context *ctdb, const char *script); -int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap); +int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap, client_async_callback fail_callback, void *callback_data); int32_t ctdb_control_tcp_client(struct ctdb_context *ctdb, uint32_t client_id, TDB_DATA indata); @@ -1279,8 +1281,6 @@ bool ctdb_stopped_monitoring(struct ctdb_context *ctdb); int ctdb_set_child_logging(struct ctdb_context *ctdb); void ctdb_lockdown_memory(struct ctdb_context *ctdb); -typedef void (*client_async_callback)(struct ctdb_context *ctdb, uint32_t node_pnn, int32_t res, TDB_DATA outdata, void *callback_data); - struct client_async_data { enum ctdb_controls opcode; bool dont_log_errors; diff --git a/packaging/RPM/ctdb.spec.in b/packaging/RPM/ctdb.spec.in index 3ded827..de8193d 100644 --- a/packaging/RPM/ctdb.spec.in +++ b/packaging/RPM/ctdb.spec.in @@ -3,7 +3,7 @@ Name: ctdb Summary: Clustered TDB Vendor: Samba Team Packager: Samba Team <sa...@samba.org> -Version: 1.2.54 +Version: 1.2.55 Release: 1GITHASH Epoch: 0 License: GNU GPL version 3 @@ -146,6 +146,12 @@ development libraries for ctdb %changelog +* Tue Nov 27 2012 : Version 1.2.55 + - Do not ignore timed out monitor events + - Track nodes that fail takeover run and ban them + - For new interfaces mark the link up by default after startup + - In 10.interface eventscript use configured interaces to monitor + - Do not set the initial interface when adding new public address * Wed Oct 30 2012 : Version 1.2.54 - Protect against double free of active monitor callback state during shutdown - Remove duplicate code to set tunables from initscript diff --git a/server/ctdb_recoverd.c b/server/ctdb_recoverd.c index a6a748e..810112f 100644 --- a/server/ctdb_recoverd.c +++ b/server/ctdb_recoverd.c @@ -1347,6 +1347,21 @@ static int sync_recovery_lock_file_across_cluster(struct ctdb_recoverd *rec) /* + * this callback is called for every node that failed to execute ctdb_takeover_run() + * and set flag to re-run takeover run. + */ +static void takeover_fail_callback(struct ctdb_context *ctdb, uint32_t node_pnn, int32_t res, TDB_DATA outdata, void *callback_data) +{ + struct ctdb_recoverd *rec = talloc_get_type(callback_data, struct ctdb_recoverd); + + DEBUG(DEBUG_ERR, (__location__ " Node %u failed the takeover run. Setting it as recovery fail culprit\n", node_pnn)); + + ctdb_set_culprit(rec, node_pnn); + rec->need_takeover_run = true; +} + + +/* we are the recmaster, and recovery is needed - start a recovery run */ static int do_recovery(struct ctdb_recoverd *rec, @@ -1633,7 +1648,7 @@ static int do_recovery(struct ctdb_recoverd *rec, return -1; } rec->need_takeover_run = false; - ret = ctdb_takeover_run(ctdb, nodemap); + ret = ctdb_takeover_run(ctdb, nodemap, NULL, NULL); if (ret != 0) { DEBUG(DEBUG_ERR, (__location__ " Unable to setup public takeover addresses. ctdb_takeover_run() failed.\n")); rec->need_takeover_run = true; @@ -1952,7 +1967,7 @@ static void ctdb_rebalance_timeout(struct event_context *ev, struct timed_event DEBUG(DEBUG_NOTICE,("Rebalance all nodes that have had ip assignment changes.\n")); - ret = ctdb_takeover_run(ctdb, rec->nodemap); + ret = ctdb_takeover_run(ctdb, rec->nodemap, NULL, NULL); if (ret != 0) { DEBUG(DEBUG_ERR, (__location__ " Unable to setup public takeover addresses. ctdb_takeover_run() failed.\n")); rec->need_takeover_run = true; @@ -2099,7 +2114,7 @@ static void process_ipreallocate_requests(struct ctdb_context *ctdb, struct ctdb rec->need_takeover_run = true; } if (ret == 0) { - ret = ctdb_takeover_run(ctdb, rec->nodemap); + ret = ctdb_takeover_run(ctdb, rec->nodemap, NULL, NULL); if (ret != 0) { DEBUG(DEBUG_ERR,("Failed to reallocate addresses: ctdb_takeover_run() failed.\n")); rec->need_takeover_run = true; @@ -3469,9 +3484,18 @@ static void main_loop(struct ctdb_context *ctdb, struct ctdb_recoverd *rec, return; } - ret = ctdb_takeover_run(ctdb, nodemap); + /* If takeover run fails, then the offending nodes are + * assigned ban culprit counts. And we re-try takeover. + * If takeover run fails repeatedly, the node would get + * banned. + * + * If rec->need_takeover_run is not set to true at this + * failure, monitoring is disabled cluster-wide (via + * startrecovery eventscript) and will not get enabled. + */ + ret = ctdb_takeover_run(ctdb, nodemap, takeover_fail_callback, rec); if (ret != 0) { - DEBUG(DEBUG_ERR, (__location__ " Unable to setup public takeover addresses. Try again later\n")); + DEBUG(DEBUG_ERR, (__location__ " Unable to setup public takeover addresses. Trying again\n")); return; } diff --git a/server/ctdb_takeover.c b/server/ctdb_takeover.c index e04ea6a..83b9347 100644 --- a/server/ctdb_takeover.c +++ b/server/ctdb_takeover.c @@ -66,7 +66,20 @@ static int ctdb_add_local_iface(struct ctdb_context *ctdb, const char *iface) CTDB_NO_MEMORY_FATAL(ctdb, i); i->name = talloc_strdup(i, iface); CTDB_NO_MEMORY(ctdb, i->name); - i->link_up = false; + /* + * If link_up defaults to true then IPs can be allocated to a + * node during the first recovery. However, then an interface + * could have its link marked down during the startup event, + * causing the IP to move almost immediately. If link_up + * defaults to false then, during normal operation, IPs added + * to a new interface can't be assigned until a monitor cycle + * has occurred and marked the new interfaces up. This makes + * IP allocation unpredictable. The following is a neat + * compromise: early in startup link_up defaults to false, so + * IPs can't be assigned, and after startup IPs can be + * assigned immediately. + */ + i->link_up = ctdb->done_startup; DLIST_ADD(ctdb->ifaces, i); @@ -922,9 +935,6 @@ static int ctdb_add_public_address(struct ctdb_context *ctdb, talloc_free(vnn); return -1; } - if (i == 0) { - vnn->iface = ctdb_find_iface(ctdb, vnn->ifaces[i]); - } } DLIST_ADD(ctdb->vnn, vnn); @@ -1974,7 +1984,8 @@ finished: /* make any IP alias changes for public addresses that are necessary */ -int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap) +int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap, + client_async_callback fail_callback, void *callback_data) { int i; struct ctdb_public_ip ip; @@ -2006,6 +2017,9 @@ int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap) async_data = talloc_zero(tmp_ctx, struct client_async_data); CTDB_NO_MEMORY_FATAL(ctdb, async_data); + async_data->fail_callback = fail_callback; + async_data->callback_data = callback_data; + for (i=0;i<nodemap->num;i++) { /* don't talk to unconnected nodes, but do talk to banned nodes */ if (nodemap->nodes[i].flags & NODE_FLAGS_DISCONNECTED) { @@ -2063,6 +2077,10 @@ int ctdb_takeover_run(struct ctdb_context *ctdb, struct ctdb_node_map *nodemap) /* tell all nodes to get their own IPs */ async_data = talloc_zero(tmp_ctx, struct client_async_data); CTDB_NO_MEMORY_FATAL(ctdb, async_data); + + async_data->fail_callback = fail_callback; + async_data->callback_data = callback_data; + for (tmp_ip=all_ips;tmp_ip;tmp_ip=tmp_ip->next) { if (tmp_ip->pnn == -1) { /* this IP won't be taken over */ @@ -2115,8 +2133,8 @@ ipreallocated: if (ctdb_client_async_control(ctdb, CTDB_CONTROL_RUN_EVENTSCRIPTS, nodes, 0, TAKEOVER_TIMEOUT(), false, data, - NULL, NULL, - NULL) != 0) { + NULL, fail_callback, + callback_data) != 0) { DEBUG(DEBUG_ERR, (__location__ " ctdb_control to updatenatgw failed\n")); } diff --git a/server/ctdb_tunables.c b/server/ctdb_tunables.c index 62a4dd4..9c70008 100644 --- a/server/ctdb_tunables.c +++ b/server/ctdb_tunables.c @@ -37,7 +37,7 @@ static const struct { { "MonitorInterval", 15, offsetof(struct ctdb_tunable, monitor_interval) }, { "TickleUpdateInterval",20, offsetof(struct ctdb_tunable, tickle_update_interval) }, { "EventScriptTimeout", 30, offsetof(struct ctdb_tunable, script_timeout) }, - { "EventScriptTimeoutCount", 1, offsetof(struct ctdb_tunable, script_timeout_count) }, + { "EventScriptTimeoutCount", 20, offsetof(struct ctdb_tunable, script_timeout_count) }, { "EventScriptUnhealthyOnTimeout", 0, offsetof(struct ctdb_tunable, script_unhealthy_on_timeout) },/* OBSOLETE */ { "RecoveryGracePeriod", 120, offsetof(struct ctdb_tunable, recovery_grace_period) }, { "RecoveryBanPeriod", 300, offsetof(struct ctdb_tunable, recovery_ban_period) }, diff --git a/server/eventscript.c b/server/eventscript.c index 2f86fdd..9ef1f3d 100644 --- a/server/eventscript.c +++ b/server/eventscript.c @@ -561,7 +561,6 @@ static void ctdb_event_script_timeout(struct event_context *ev, struct timed_eve case CTDB_EVENT_TAKE_IP: case CTDB_EVENT_RELEASE_IP: case CTDB_EVENT_STOPPED: - case CTDB_EVENT_MONITOR: case CTDB_EVENT_STATUS: state->scripts->scripts[state->current].status = 0; DEBUG(DEBUG_ERR,("Ignoring hung script for %s call %d\n", state->options, state->call)); diff --git a/tools/ctdb.c b/tools/ctdb.c index 03373a2..b6daa5a 100644 --- a/tools/ctdb.c +++ b/tools/ctdb.c @@ -1954,10 +1954,12 @@ static int control_delip(struct ctdb_context *ctdb, int argc, const char **argv) } if (ips->ips[i].pnn == options.pnn) { - ret = find_other_host_for_public_ip(ctdb, &addr); - if (ret != -1) { + int pnn; + + pnn = find_other_host_for_public_ip(ctdb, &addr); + if (pnn != -1) { do { - ret = move_ip(ctdb, &addr, ret); + ret = move_ip(ctdb, &addr, pnn); if (ret != 0) { DEBUG(DEBUG_ERR,("Failed to move ip to node %d. Wait 3 seconds and try again.\n", options.pnn)); sleep(3); -- CTDB repository