Hi, Here are the logs when pacemaker fails to start postgres service on master. It manage to start only postgres slave. I tried different configuration with pgslqms and pgsql resource agents. Those errors are when I use pgsqlms agent, which configuration I have sent in first mail:
Apr 25 16:40:23 [4213] master lrmd: info: log_execute: executing - rsc:PGSQL action:start call_id:51 launching as "postgres" command "/usr/lib/postgresql/9.5/bin/pg_ctl --pgdata /var/lib/postgresql/9.5/main -w --timeout 120 start -o -c config_file=/etc/postgresql/9.5/main/postgresql.conf" Apr 25 16:40:24 [4211] master cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='PGSQL']/lrm_rsc_op[@id='PGSQL_last_0']: @operation_key=PGSQL_start_0, @operation=start, @transition-key=12:30:0:078c2b66-b095-49c4-947b-2427dd7852bf, @transition-magic=0:0;12:30:0:078c2b66-b095-49c4-947b-2427dd7852bf, @call-id=176, @rc-code=0, @exec-time=1146, @queue-time=0 Apr 25 16:40:53 [4216] master crmd: debug: crm_timer_start: Started Shutdown Escalation (I_STOP:1200000ms), src=53 Apr 25 16:41:23 [4213] master lrmd: warning: child_timeout_callback: PGSQL_start_0 process (PID 5986) timed out Part of the log is attached. On Tue, 23 Apr 2019 at 17:28, Danka Ivanović <danka.ivano...@gmail.com> wrote: > Hi, > It seems that ldap timeout caused cluster failure. Cluster is checking > status every 15s on master and 16s on slave. Cluster needs postgres user > for authentication, but ldap first query user on ldap server and then > localy on host. When connection to ldap server was interrupted, cluster > couldn't find postgres user and authenticate on db to check state. Problem > is solved with reconfiguring /etc/ldap.conf and /etc/nslcd.conf. Following > variable is added: nss_initgroups_ignoreusers with specified local users > which should be ignored when querying ldap server. Thanks for your help. :) > Another problem is that I cannot start postgres master with pacemaker. > When I start postgres manually (with systemd) and then start pacemaker on > slave, pacemaker is able to recognize master and start slave and failover > works. > That is another problem which I didn't manage to solve. Should I send a > new mail for that issue or we can continue in this thread? > > On Fri, 19 Apr 2019 at 19:19, Jehan-Guillaume de Rorthais <j...@dalibo.com> > wrote: > >> On Fri, 19 Apr 2019 17:26:14 +0200 >> Danka Ivanović <danka.ivano...@gmail.com> wrote: >> ... >> > Should I change any of those timeout parameters in order to avoid >> timeout? >> >> You can try to raise the timeout, indeed. But as far as we don't know >> **why** >> your VMs froze for some time, it is difficult to guess how high should be >> these timeouts. >> >> Not to mention that it will raise your RTO. >> > > > -- > Pozdrav > Danka Ivanovic > -- Pozdrav Danka Ivanovic
Apr 25 16:39:50 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444e8e80 for uid=0 gid=0 pid=5791 id=c93d535d-77d8-4556-9a63-d9a1c2b45de9 Apr 25 16:39:50 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5791-13) Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5791] Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5791-13) Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5791-13) state:2 Apr 25 16:39:50 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5791-13-header Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5791-13-header Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5791-13-header Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5837-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5837] Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5837-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5837-24) state:2 Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9) Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-15544-5837-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-15544-5837-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-15544-5837-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5837-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5837] Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5837-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5837-24) state:2 Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9) Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-response-15544-5837-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-event-15544-5837-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-request-15544-5837-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5837-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5837] Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5837-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5837-24) state:2 Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9) Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-15544-5837-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-15544-5837-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-15544-5837-24-header Apr 25 16:39:50 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444e8e80 for uid=0 gid=0 pid=5841 id=92b5b3a5-7575-4ed9-a26b-5877b2ab1bab Apr 25 16:39:50 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5841-13) Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5841] Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5841-13) Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5841-13) state:2 Apr 25 16:39:50 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5841-13-header Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5841-13-header Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5841-13-header Apr 25 16:39:50 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444e8e80 for uid=0 gid=0 pid=5847 id=6f50e4fb-5cb0-43c9-98f8-15c3cacc7102 Apr 25 16:39:50 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5847-13) Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5847] Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: info: cib_process_request: Forwarding cib_apply_diff operation for section 'all' to master (origin=local/cibadmin/2) Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:39:50 [4211] master cib: debug: xml_patch_version_check: Can apply patch 5.758.3 to 5.758.2 Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: --- 5.758.2 2 Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: +++ 5.759.0 (null) Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: + /cib: @epoch=759, @num_updates=0 Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ /cib/configuration/resources: <primitive id="PGSQL" class="ocf" provider="heartbeat" type="pgsqlms"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <instance_attributes id="PGSQL-instance_attributes"> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <nvpair name="pgdata" value="/var/lib/postgresql/9.5/main" id="PGSQL-instance_attributes-pgdata"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <nvpair name="bindir" value="/usr/lib/postgresql/9.5/bin" id="PGSQL-instance_attributes-bindir"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <nvpair name="pghost" value="/var/run/postgresql/" id="PGSQL-instance_attributes-pghost"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <nvpair name="recovery_template" value="/etc/postgresql/9.5/main/recovery.conf.pcmk" id="PGSQL-instance_attributes-recovery_template"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <nvpair name="start_opts" value="-c config_file=/etc/postgresql/9.5/main/postgresql.conf" id="PGSQL-instance_attributes-start_opts"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ </instance_attributes> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <operations> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="start" timeout="60s" interval="0" id="PGSQL-start-0"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="stop" timeout="60s" interval="0" id="PGSQL-stop-0"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="promote" timeout="15s" interval="0" id="PGSQL-promote-0"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="demote" timeout="120s" interval="0" id="PGSQL-demote-0"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="monitor" interval="15s" timeout="10s" role="Master" id="PGSQL-monitor-15s"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="monitor" interval="16s" timeout="10s" role="Slave" id="PGSQL-monitor-16s"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="notify" timeout="60" interval="0" id="PGSQL-notify-0"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ </operations> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ </primitive> Apr 25 16:39:50 [4211] master cib: debug: activateCibXml: Triggering CIB write for cib_apply_diff op Apr 25 16:39:50 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.759.0 to 5.758.2 Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: STONITH timeout: 60000 Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: STONITH of failed nodes is enabled Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: Stop all active resources: false Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: Cluster is symmetric - resources can run anywhere by default Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: Default stickiness: 0 Apr 25 16:39:50 [4212] master stonith-ng: notice: unpack_config: On loss of CCM Quorum: Ignore Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Apr 25 16:39:50 [4212] master stonith-ng: debug: cib_device_update: Device fencing-test-rsyslog is allowed on master: score=0 Apr 25 16:39:50 [4212] master stonith-ng: debug: stonith_device_register: Device 'fencing-test-rsyslog' already existed in device list (1 active devices) Apr 25 16:39:50 [4211] master cib: info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=master/cibadmin/2, version=5.759.0) Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5847-13) Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5847-13) state:2 Apr 25 16:39:50 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5847-13-header Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5847-13-header Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5847-13-header Apr 25 16:39:50 [4211] master cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-11.raw Apr 25 16:39:50 [4211] master cib: debug: cib_file_write_with_digest: Writing CIB to disk Apr 25 16:39:50 [4213] master lrmd: debug: process_lrmd_message: Processed lrmd_rsc_info operation from a0d15606-efa1-42dc-8892-885bfe52b70f: rc=0, reply=0, notify=0, exit=-439087056 Apr 25 16:39:50 [4213] master lrmd: debug: process_lrmd_message: Processed lrmd_rsc_register operation from a0d15606-efa1-42dc-8892-885bfe52b70f: rc=0, reply=1, notify=1, exit=-439087056 Apr 25 16:39:50 [4213] master lrmd: debug: process_lrmd_message: Processed lrmd_rsc_info operation from a0d15606-efa1-42dc-8892-885bfe52b70f: rc=0, reply=0, notify=0, exit=-439087056 Apr 25 16:39:50 [4213] master lrmd: debug: process_lrmd_message: Processed lrmd_rsc_exec operation from a0d15606-efa1-42dc-8892-885bfe52b70f: rc=43, reply=1, notify=0, exit=-439087056 Apr 25 16:39:50 [4213] master lrmd: debug: log_execute: executing - rsc:PGSQL action:monitor call_id:43 Apr 25 16:39:50 [4211] master cib: info: cib_file_write_with_digest: Wrote version 5.759.0 of the CIB to disk (digest: 16f02c16f467c9ef3114138a3febd8c5) Apr 25 16:39:50 [4211] master cib: debug: cib_file_write_with_digest: Wrote digest 16f02c16f467c9ef3114138a3febd8c5 to disk Apr 25 16:39:50 [4211] master cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.G7Oz1H (digest: /var/lib/pacemaker/cib/cib.VFtf2I) Apr 25 16:39:50 [4211] master cib: debug: cib_file_write_with_digest: Activating /var/lib/pacemaker/cib/cib.G7Oz1H Apr 25 16:39:50 [5856] master crm_node: info: get_cluster_type: Verifying cluster type: 'corosync' Apr 25 16:39:50 [5856] master crm_node: info: get_cluster_type: Assuming an active 'corosync' cluster Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5856-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5856] Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5856] master crm_node: debug: qb_ipcc_disconnect: qb_ipcc_disconnect() Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-request-15544-5856-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5856-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5856-24) state:2 Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9) Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-response-15544-5856-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-response-15544-5856-24-header Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-event-15544-5856-24-header Apr 25 16:39:50 [5856] master crm_node: debug: get_local_nodeid: Local nodeid is 1 Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-event-15544-5856-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-request-15544-5856-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5856-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5856] Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5856] master crm_node: debug: qb_ipcc_disconnect: qb_ipcc_disconnect() Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-request-15544-5856-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5856-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5856-24) state:2 Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9) Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-15544-5856-24-header Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-response-15544-5856-24-header Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-event-15544-5856-24-header Apr 25 16:39:50 [5856] master crm_node: info: crm_xml_cleanup: Cleaning up memory from libxml2 Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-15544-5856-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-15544-5856-24-header Apr 25 16:39:50 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5858 id=97335234-7d26-4466-a47a-fb474c95ac16 Apr 25 16:39:50 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5858-13) Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5858] Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:39:50 [5858] master crm_resource: debug: cib_native_signon_raw: Connection to CIB successful Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: --- 5.759.0 2 Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: +++ 5.759.1 (null) Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: + /cib: @num_updates=1 Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="PGSQL" type="pgsqlms" class="ocf" provider="heartbeat"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <lrm_rsc_op id="PGSQL_last_failure_0" operation_key="PGSQL_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="5:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" transition-magic="0:5;5:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" exit-reason="You must set meta parameter notify=true for your master resou Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <lrm_rsc_op id="PGSQL_last_0" operation_key="PGSQL_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="5:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" transition-magic="0:5;5:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" exit-reason="You must set meta parameter notify=true for your master resource" on_ Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ </lrm_resource> Apr 25 16:39:50 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.759.1 to 5.759.0 Apr 25 16:39:50 [4211] master cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=secondary/crmd/277, version=5.759.1) Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: STONITH timeout: 60000 Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: STONITH of failed nodes is enabled Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: Stop all active resources: false Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: Cluster is symmetric - resources can run anywhere by default Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: Default stickiness: 0 Apr 25 16:39:50 [5858] master crm_resource: notice: unpack_config: On loss of CCM Quorum: Ignore Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Apr 25 16:39:50 [5858] master crm_resource: info: determine_online_status_fencing: Node master is active Apr 25 16:39:50 [5858] master crm_resource: info: determine_online_status: Node master is online Apr 25 16:39:50 [5858] master crm_resource: debug: populate_hash: Overwriting value of master-PGSQL: -1000 -> -INFINITY Apr 25 16:39:50 [5858] master crm_resource: info: determine_online_status_fencing: Node secondary is active Apr 25 16:39:50 [5858] master crm_resource: info: determine_online_status: Node secondary is online Apr 25 16:39:50 [5858] master crm_resource: debug: determine_op_status: AWSVIP_monitor_0 on master returned 'ok' (0) instead of the expected value: 'not running' (7) Apr 25 16:39:50 [5858] master crm_resource: info: determine_op_status: Operation monitor found resource AWSVIP active on master Apr 25 16:39:50 [5858] master crm_resource: debug: determine_op_status: AWSVIP_monitor_0 on master returned 'ok' (0) instead of the expected value: 'not running' (7) Apr 25 16:39:50 [5858] master crm_resource: info: determine_op_status: Operation monitor found resource AWSVIP active on master Apr 25 16:39:50 [5858] master crm_resource: debug: cli_resource_print_attribute: Looking up notify in PGSQL Apr 25 16:39:50 [5858] master crm_resource: debug: cib_native_signoff: Signing out of the CIB Service Apr 25 16:39:50 [5858] master crm_resource: debug: qb_ipcc_disconnect: qb_ipcc_disconnect() Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5858-13-header Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5858-13-header Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5858-13) Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5858-13) state:2 Apr 25 16:39:50 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5858-13-header Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5858-13-header Apr 25 16:39:50 [5858] master crm_resource: info: crm_xml_cleanup: Cleaning up memory from libxml2 Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5858-13-header Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5858-13-header pgsqlms(PGSQL)[5849]: 2019/04/25_16:39:50 ERROR: You must set meta parameter notify=true for your master resource Apr 25 16:39:50 [4213] master lrmd: debug: operation_finished: PGSQL_monitor_0:5849 - exited with rc=5 Apr 25 16:39:50 [4213] master lrmd: notice: operation_finished: PGSQL_monitor_0:5849:stderr [ ocf-exit-reason:You must set meta parameter notify=true for your master resource ] Apr 25 16:39:50 [4213] master lrmd: debug: operation_finished: PGSQL_monitor_0:5849:stdout [ -- empty -- ] Apr 25 16:39:50 [4213] master lrmd: debug: log_finished: finished - rsc:PGSQL action:monitor call_id:43 pid:5849 exit-code:5 exec-time:91ms queue-time:0ms Apr 25 16:39:50 [4216] master crmd: debug: create_operation_update: do_update_resource: Updating resource PGSQL after monitor op complete (interval=0) Apr 25 16:39:50 [5866] master crm_node: info: get_cluster_type: Verifying cluster type: 'corosync' Apr 25 16:39:50 [5866] master crm_node: info: get_cluster_type: Assuming an active 'corosync' cluster Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5866-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5866] Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5866] master crm_node: debug: qb_ipcc_disconnect: qb_ipcc_disconnect() Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-request-15544-5866-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5866-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5866-24) state:2 Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9) Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-response-15544-5866-24-header Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-response-15544-5866-24-header Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-event-15544-5866-24-header Apr 25 16:39:50 [5866] master crm_node: debug: get_local_nodeid: Local nodeid is 1 Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-event-15544-5866-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-request-15544-5866-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5866-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5866] Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168 Apr 25 16:39:50 [5866] master crm_node: debug: qb_ipcc_disconnect: qb_ipcc_disconnect() Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-request-15544-5866-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5866-24) Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5866-24) state:2 Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9) Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-15544-5866-24-header Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-response-15544-5866-24-header Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-event-15544-5866-24-header Apr 25 16:39:50 [5866] master crm_node: info: crm_xml_cleanup: Cleaning up memory from libxml2 Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-15544-5866-24-header Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-15544-5866-24-header Apr 25 16:39:50 [4216] master crmd: info: action_synced_wait: Managed pgsqlms_meta-data_0 process 5859 exited with rc=0 Apr 25 16:39:50 [4216] master crmd: notice: process_lrm_event: Operation PGSQL_monitor_0: not installed (node=master, call=43, rc=5, cib-update=40, confirmed=true) Apr 25 16:39:50 [4216] master crmd: notice: process_lrm_event: master-PGSQL_monitor_0:43 [ ocf-exit-reason:You must set meta parameter notify=true for your master resource\n ] Apr 25 16:39:50 [4216] master crmd: debug: update_history_cache: Updating history for 'PGSQL' with monitor op Apr 25 16:39:50 [4211] master cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/40) Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: --- 5.759.1 2 Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: +++ 5.759.2 (null) Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: + /cib: @num_updates=2 Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="PGSQL" type="pgsqlms" class="ocf" provider="heartbeat"/> Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <lrm_rsc_op id="PGSQL_last_failure_0" operation_key="PGSQL_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="4:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" transition-magic="0:5;4:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" exit-reason="You must set meta parameter notify=true for your master resou Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <lrm_rsc_op id="PGSQL_last_0" operation_key="PGSQL_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="4:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" transition-magic="0:5;4:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" exit-reason="You must set meta parameter notify=true for your master resource" on_ Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ </lrm_resource> Apr 25 16:39:50 [4211] master cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=master/crmd/40, version=5.759.2) Apr 25 16:39:50 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.759.2 to 5.759.1 Apr 25 16:39:55 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:03 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444eda80 for uid=0 gid=0 pid=5871 id=77319e05-d144-4d61-a6a1-7b02ce9d7eaa Apr 25 16:40:03 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5871-13) Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5871] Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:03 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5871-13) Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5871-13) state:2 Apr 25 16:40:03 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5871-13-header Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5871-13-header Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5871-13-header Apr 25 16:40:03 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444eda80 for uid=0 gid=0 pid=5881 id=c3aae380-3d8d-4800-a167-7d0f676010c8 Apr 25 16:40:03 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5881-13) Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5881] Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:03 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5881-13) Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5881-13) state:2 Apr 25 16:40:03 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5881-13-header Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5881-13-header Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5881-13-header Apr 25 16:40:03 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444eda80 for uid=0 gid=0 pid=5887 id=eb5ba3d7-d898-4a55-988a-ba46894ff6b8 Apr 25 16:40:03 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5887-13) Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5887] Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:03 [4211] master cib: info: cib_process_request: Forwarding cib_apply_diff operation for section 'all' to master (origin=local/cibadmin/2) Apr 25 16:40:03 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:03 [4211] master cib: debug: xml_patch_version_check: Can apply patch 5.759.3 to 5.759.2 Apr 25 16:40:03 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: Diff: --- 5.759.2 2 Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: Diff: +++ 5.760.0 771ea8eb8446b4c081f2915124d06d63 Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: -- /cib/configuration/resources/primitive[@id='PGSQL'] Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: + /cib: @epoch=760, @num_updates=0 Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ /cib/configuration/resources: <master id="PGSQL-HA"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <meta_attributes id="PGSQL-HA-meta_attributes"> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="master-max" value="1" id="PGSQL-HA-meta_attributes-master-max"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="master-node-max" value="1" id="PGSQL-HA-meta_attributes-master-node-max"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="clone-max" value="2" id="PGSQL-HA-meta_attributes-clone-max"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="clone-node-max" value="1" id="PGSQL-HA-meta_attributes-clone-node-max"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="notify" value="true" id="PGSQL-HA-meta_attributes-notify"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="interleave" value="true" id="PGSQL-HA-meta_attributes-interleave"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ </meta_attributes> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <primitive id="PGSQL" class="ocf" provider="heartbeat" type="pgsqlms"> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <instance_attributes id="PGSQL-instance_attributes"> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="pgdata" value="/var/lib/postgresql/9.5/main" id="PGSQL-instance_attributes-pgdata"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="bindir" value="/usr/lib/postgresql/9.5/bin" id="PGSQL-instance_attributes-bindir"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="pghost" value="/var/run/postgresql/" id="PGSQL-instance_attributes-pghost"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="recovery_template" value="/etc/postgresql/9.5/main/recovery.conf.pcmk" id="PGSQL-instance_attributes-recovery_template"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="start_opts" value="-c config_file=/etc/postgresql/9.5/main/postgresql.conf" id="PGSQL-instance_attributes-start_opts"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ </instance_attributes> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <operations> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="start" timeout="60s" interval="0" id="PGSQL-start-0"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="stop" timeout="60s" interval="0" id="PGSQL-stop-0"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="promote" timeout="15s" interval="0" id="PGSQL-promote-0"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="demote" timeout="120s" interval="0" id="PGSQL-demote-0"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="monitor" interval="15s" timeout="10s" role="Master" id="PGSQL-monitor-15s"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="monitor" interval="16s" timeout="10s" role="Slave" id="PGSQL-monitor-16s"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="notify" timeout="60" interval="0" id="PGSQL-notify-0"/> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ </operations> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ </primitive> Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ </master> Apr 25 16:40:03 [4211] master cib: debug: activateCibXml: Triggering CIB write for cib_apply_diff op Apr 25 16:40:03 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.760.0 to 5.759.2 Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: STONITH timeout: 60000 Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: STONITH of failed nodes is enabled Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: Stop all active resources: false Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: Cluster is symmetric - resources can run anywhere by default Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: Default stickiness: 0 Apr 25 16:40:03 [4212] master stonith-ng: notice: unpack_config: On loss of CCM Quorum: Ignore Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Apr 25 16:40:03 [4212] master stonith-ng: debug: cib_device_update: Device fencing-test-rsyslog is allowed on master: score=0 Apr 25 16:40:03 [4212] master stonith-ng: debug: stonith_device_register: Device 'fencing-test-rsyslog' already existed in device list (1 active devices) Apr 25 16:40:03 [4211] master cib: info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=master/cibadmin/2, version=5.760.0) Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5887-13) Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5887-13) state:2 Apr 25 16:40:03 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5887-13-header Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5887-13-header Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5887-13-header Apr 25 16:40:03 [4211] master cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-12.raw Apr 25 16:40:03 [4211] master cib: debug: cib_file_write_with_digest: Writing CIB to disk Apr 25 16:40:03 [4211] master cib: info: cib_file_write_with_digest: Wrote version 5.760.0 of the CIB to disk (digest: 85843593b18ef9bd7a6b33630ead2936) Apr 25 16:40:03 [4211] master cib: debug: cib_file_write_with_digest: Wrote digest 85843593b18ef9bd7a6b33630ead2936 to disk Apr 25 16:40:03 [4211] master cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.N7t074 (digest: /var/lib/pacemaker/cib/cib.FeFUqC) Apr 25 16:40:03 [4211] master cib: debug: cib_file_write_with_digest: Activating /var/lib/pacemaker/cib/cib.N7t074 Apr 25 16:40:04 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5893 id=21032fff-7a5b-4e0f-86b9-f8cd87ffa267 Apr 25 16:40:04 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5893-13) Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5893] Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:04 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5893-13) Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5893-13) state:2 Apr 25 16:40:04 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5893-13-header Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5893-13-header Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5893-13-header Apr 25 16:40:04 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5903 id=b145010e-eaf7-40f0-a699-b5a4a0dbdad1 Apr 25 16:40:04 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5903-13) Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5903] Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:04 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5903-13) Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5903-13) state:2 Apr 25 16:40:04 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5903-13-header Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5903-13-header Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5903-13-header Apr 25 16:40:04 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5909 id=527ca328-9436-4720-b202-2a82d415cafb Apr 25 16:40:04 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5909-13) Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5909] Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:04 [4211] master cib: info: cib_process_request: Forwarding cib_apply_diff operation for section 'all' to master (origin=local/cibadmin/2) Apr 25 16:40:04 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:04 [4211] master cib: debug: xml_patch_version_check: Can apply patch 5.760.1 to 5.760.0 Apr 25 16:40:04 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:04 [4211] master cib: info: cib_perform_op: Diff: --- 5.760.0 2 Apr 25 16:40:04 [4211] master cib: info: cib_perform_op: Diff: +++ 5.761.0 (null) Apr 25 16:40:04 [4211] master cib: info: cib_perform_op: + /cib: @epoch=761 Apr 25 16:40:04 [4211] master cib: info: cib_perform_op: ++ /cib/configuration/constraints: <rsc_colocation id="IPAWSIP-WITH-MASTER" score="INFINITY" rsc="AWSVIP" with-rsc="PGSQL-HA" with-rsc-role="Master"/> Apr 25 16:40:04 [4211] master cib: debug: activateCibXml: Triggering CIB write for cib_apply_diff op Apr 25 16:40:04 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.761.0 to 5.760.0 Apr 25 16:40:04 [4211] master cib: info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=master/cibadmin/2, version=5.761.0) Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5909-13) Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5909-13) state:2 Apr 25 16:40:04 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5909-13-header Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5909-13-header Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5909-13-header Apr 25 16:40:04 [4211] master cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-13.raw Apr 25 16:40:04 [4211] master cib: debug: cib_file_write_with_digest: Writing CIB to disk Apr 25 16:40:04 [4211] master cib: info: cib_file_write_with_digest: Wrote version 5.761.0 of the CIB to disk (digest: cbf371ff24027c60389dd1df088dda45) Apr 25 16:40:04 [4216] master crmd: debug: do_lrm_rsc_op: Stopped 0 recurring operations in preparation for AWSVIP_stop_0 Apr 25 16:40:04 [4213] master lrmd: debug: process_lrmd_message: Processed lrmd_rsc_exec operation from a0d15606-efa1-42dc-8892-885bfe52b70f: rc=44, reply=1, notify=0, exit=-439087056 Apr 25 16:40:04 [4213] master lrmd: info: log_execute: executing - rsc:AWSVIP action:stop call_id:44 Apr 25 16:40:04 [4211] master cib: debug: cib_file_write_with_digest: Wrote digest cbf371ff24027c60389dd1df088dda45 to disk Apr 25 16:40:04 [4211] master cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.8CQj3j (digest: /var/lib/pacemaker/cib/cib.IpBJgU) Apr 25 16:40:04 [4211] master cib: debug: cib_file_write_with_digest: Activating /var/lib/pacemaker/cib/cib.8CQj3j Apr 25 16:40:06 [4216] master crmd: debug: throttle_cib_load: cib load: 0.001000 (3 ticks in 30s) Apr 25 16:40:06 [4216] master crmd: debug: throttle_load_avg: Current load is 0.040000 (full: 0.04 0.07 0.02 1/229 5948) Apr 25 16:40:06 [4216] master crmd: debug: throttle_io_load: Current IO load is 0.000000 Apr 25 16:40:07 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5960 id=c11cc076-8c66-41f6-abdf-a980e63a8e1e Apr 25 16:40:07 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5960-13) Apr 25 16:40:07 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5960] Apr 25 16:40:07 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:07 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:07 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:07 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:07 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5960-13) Apr 25 16:40:07 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5960-13) state:2 Apr 25 16:40:07 [4211] master cib: debug: crm_client_destroy: Destroying 0 events Apr 25 16:40:07 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_ro-response-4211-5960-13-header Apr 25 16:40:07 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_ro-event-4211-5960-13-header Apr 25 16:40:07 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_ro-request-4211-5960-13-header Apr 25 16:40:09 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled awsvip(AWSVIP)[5911]: 2019/04/25_16:40:10 INFO: secondary_private_ip has been successfully brought down (10.x.x.x) awsvip(AWSVIP)[5911]: 2019/04/25_16:40:10 DEBUG: AWSVIP stop : 0 Apr 25 16:40:10 [4213] master lrmd: debug: operation_finished: AWSVIP_stop_0:5911 - exited with rc=0 Apr 25 16:40:10 [4213] master lrmd: debug: operation_finished: AWSVIP_stop_0:5911:stderr [ -- empty -- ] Apr 25 16:40:10 [4213] master lrmd: debug: operation_finished: AWSVIP_stop_0:5911:stdout [ -- empty -- ] Apr 25 16:40:10 [4213] master lrmd: info: log_finished: finished - rsc:AWSVIP action:stop call_id:44 pid:5911 exit-code:0 exec-time:6302ms queue-time:0ms Apr 25 16:40:10 [4216] master crmd: debug: create_operation_update: do_update_resource: Updating resource AWSVIP after stop op complete (interval=0) Apr 25 16:40:10 [4216] master crmd: notice: process_lrm_event: Operation AWSVIP_stop_0: ok (node=master, call=44, rc=0, cib-update=41, confirmed=true) Apr 25 16:40:10 [4216] master crmd: debug: update_history_cache: Updating history for 'AWSVIP' with stop op Apr 25 16:40:10 [4211] master cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/41) Apr 25 16:40:10 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:10 [4211] master cib: info: cib_perform_op: Diff: --- 5.761.0 2 Apr 25 16:40:10 [4211] master cib: info: cib_perform_op: Diff: +++ 5.761.1 (null) Apr 25 16:40:10 [4211] master cib: info: cib_perform_op: + /cib: @num_updates=1 Apr 25 16:40:10 [4211] master cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='AWSVIP']/lrm_rsc_op[@id='AWSVIP_last_0']: @operation_key=AWSVIP_stop_0, @operation=stop, @transition-key=8:28:0:078c2b66-b095-49c4-947b-2427dd7852bf, @transition-magic=0:0;8:28:0:078c2b66-b095-49c4-947b-2427dd7852bf, @call-id=44, @last-run=1556210404, @last-rc-change=1556210404, @exec-time=6302 Apr 25 16:40:10 [4211] master cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=master/crmd/41, version=5.761.1) Apr 25 16:40:10 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.761.1 to 5.761.0 Apr 25 16:40:15 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:23 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5971 id=6d820cd5-f375-4af0-b4a8-99ee56f25447 Apr 25 16:40:23 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5971-13) Apr 25 16:40:23 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5971] Apr 25 16:40:23 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:23 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:23 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096 Apr 25 16:40:23 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled Apr 25 16:40:23 [4216] master crmd: debug: crm_client_new: Connecting 0x559525e82730 for uid=0 gid=0 pid=5971 id=68c1b0cc-17a2-4013-924f-f29860582058 Apr 25 16:40:23 [4216] master crmd: debug: handle_new_connection: IPC credentials authenticated (4216-5971-13) Apr 25 16:40:23 [4216] master crmd: debug: qb_ipcs_shm_connect: connecting to client [5971] Apr 25 16:40:23 [4216] master crmd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792 Apr 25 16:40:23 [4216] master crmd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792 Apr 25 16:40:23 [4216] master crmd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792 Apr 25 16:40:23 [4214] master attrd: debug: crm_client_new: Connecting 0x55d565951a50 for uid=0 gid=0 pid=5971 id=fdb1c5ce-ecd1-4f8e-ac74-07b7d7989c29 Apr 25 16:40:23 [4214] master attrd: debug: handle_new_connection: IPC credentials authenticated (4214-5971-10) Apr 25 16:40:23 [4214] master attrd: debug: qb_ipcs_shm_connect: connecting to client [5971] Apr 25 16:40:23 [4214] master attrd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792 Apr 25 16:40:23 [4214] master attrd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792 Apr 25 16:40:23 [4214] master attrd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792 Apr 25 16:40:23 [4214] master attrd: debug: attrd_client_update: Broadcasting fail-count-PGSQL[master] = (null) Apr 25 16:40:23 [4211] master cib: info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='master']//lrm_resource[@id='PGSQL'] to master (origin=local/crmd/42)
_______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/