On 12-08-31 04:16 AM, Knut Ingvald Dietzel wrote: > Hello, > > In a four node cluster, one master (origin & provider) and three slaves > (consumers), I have an issue related to failover. > > Using the slonik_failover.sh script (originally generated by altperl tools, I > suspect) to manually fail the master node (id 1) and make one of the slaves > master (id 4) fails. > > The script is executed on node 4: > > ./slonik_failover.sh host01-master 5432 host04-slave 5432 haisa haisa \ > haisa_cluster replicator 1 4 > > Extract from the log: > > PL/pgSQL function "failednode2" line 14 at SQL statement > NOTICE: executing "_haisa_cluster".failedNode2 on node 4 > <stdin>:1: waiting for event (1,5004362283). node 2 only on event > 5004362281, node 3 only on event 5004362281 > <stdin>:1: waiting for event (1,5004362283). node 2 only on event > 5004362281, node 3 only on event 5004362281 > NOTICE: executing "_haisa_cluster".failedNode2 on node 4 > <stdin>:1: PGRES_FATAL_ERROR lock table "_haisa_cluster".sl_event_lock, > "_haisa_cluster".sl_config_lock;select > "_haisa_cluster".failedNode2(1,4,27,'5004362261','5004362284'); - ERROR: > duplicate key value violates unique constraint "sl_event-pkey" > DETAIL: Key (ev_origin, ev_seqno)=(1, 5004362284) already exists. > CONTEXT: SQL statement "insert into "_haisa_cluster".sl_event > (ev_origin, ev_seqno, ev_timestamp, > ev_snapshot, > ev_type, ev_data1, ev_data2, ev_data3) > values > (p_failed_node, p_ev_seqfake, CURRENT_TIMESTAMP, > v_row.ev_snapshot, > 'FAILOVER_SET', p_failed_node::text, p_backup_node::text, > p_set_id::text)" > > From what I have been able to find out so far, slonik should wait for > the slon engine to restart, and then call failedNode2() on the node with > the highest SYNC. Though, from the log above failedNode2() appears to > be called twice, the second instance fails in getting lock, and the > process of failing over node 1 to 4 fails. > > Firstly, is my interpretation in the vicinity of being correct? >
When Slonik (<=2.1.x) does a fail over it generates a 'fake' FAILOVER event using a ev_origin=$failed_node with the highest sequence number it can see of that failed node. It pushes this event into sl_event on one of the remaining nodes. In the test case you describe it sounds like that slon is still running on the failed node. Slony <=2.1.x have numerous race conditions with failover one of the ones I've seen is where a 'real' SYNC event ie 1,1234 that escaped from the failed node can conflict with the faked FAILOVER event 1,1234. I rewrote a lot of the failover logic in 2.2 to try to address many of these issues. It should do a much better job at waiting for slons to restart etc... 2.2 is still beta and I wouldn't recommend it for production use yet but I encourage you to look at it to see if it addresses your issues. > Secondly, I am unsure with regard to how to debug this further. Any, advice > would be much appreciated! > > This is with Slony1 2.1.1 and PostgreSQL 9.1.5 on Red Hat Enterprise Linux > Server release 6.3 (Santiago) x86_64. > > > > > _______________________________________________ > Slony1-general mailing list > [email protected] > http://lists.slony.info/mailman/listinfo/slony1-general > _______________________________________________ Slony1-general mailing list [email protected] http://lists.slony.info/mailman/listinfo/slony1-general
