Hello,
I have now moved on to creating a cluster. I’m following the instructions
here: https://packetfence.org/doc/PacketFence_Clustering_Guide.html
The cluster is distributed; each of the 3 nodes are in different layer 3
networks.
I’ve reached section 3.5.3 MariaDB sync in the instructions and it is not
succeeding.
On the master node, I have executed /usr/local/pf/sbin/pf-mariadb
-force-new-cluster. The mysqld process is running and the following errors
occur in /usr/local/pf/logs/packetfence.log:
Oct 16 12:12:27 nadc1-pfence-01 packetfence: -e(23457) FATAL: unable to connect
to database: Can't connect to MySQL server on '127.0.0.1' (111 "Connection
refused") at /usr/local/pf/lib/pf/dal.pm line 70.
I find the database is running on the master node. I can successfully connect
using the command ‘mysql -u root -p’. I can also connect using the command
‘mysql -h localhost -u root -p’. And, from the secondary node, I can connect
to the database on the master node using the command ‘mysql -h <remotehostIP>
-u root -p’. I’m confident the database is running and listening.
On the other node, the configuration is synced, haproxy-db and
httpd.webservices are running. On all servers, iptables is stopped. The
MariaDB data is flushed using ‘rm -fr /var/lib/mysql/*’ and I can see the
folder is empty.
I execute the command ‘systemctl restart packetfence-mariadb’ and see the
following message repeating in /usr/local/pf/logs/packetfence.log:
Oct 16 12:01:46 novi-pfence-01 packetfence: pf-mariadb(59988) INFO: There is a
peer with an alive DB. Will attempt to connect to the cluster
(main::startup_clean_shutdown)
Using the command ‘systemctl status packetfence-mariadb’ a couple times in a
row, the following errors occur showing that the process is crashing and
restarting.
● packetfence-mariadb.service - PacketFence MariaDB instance
Loaded: loaded (/lib/systemd/system/packetfence-mariadb.service; enabled;
vendor preset: enabled)
Active: activating (start) since Fri 2020-10-16 12:26:23 EDT; 34s ago
Process: 215522 ExecStartPre=/usr/local/pf/bin/pfcmd generatemariadbconfig
(code=exited, status=0/SUCCESS)
Main PID: 215534 (pf-mariadb)
Status: "Free innodb buffer pool"
Tasks: 4 (limit: 36864)
CGroup: /packetfence.slice/packetfence-base.slice/packetfence-mariadb.service
├─215534 pf-mariadb
├─218833 /bin/bash /usr/bin/mysqld_safe
--defaults-file=/usr/local/pf/var/conf/mariadb.cnf
├─218981 /usr/sbin/mysqld
--defaults-file=/usr/local/pf/var/conf/mariadb.cnf --basedir=/usr
--datadir=/var/lib/mysql --plugin-dir=/usr/lib/x86_64-linux-gnu/mari
└─218982 logger -t mysqld -p daemon error
Oct 16 12:26:54 novi-pfence-01 mysqld[218781]: 2020-10-16 12:26:54
139773373455744 [Note] Recovering after a crash using tc.log
Oct 16 12:26:54 novi-pfence-01 mysqld[218781]: 2020-10-16 12:26:54
139773373455744 [Note] Starting crash recovery...
Oct 16 12:26:54 novi-pfence-01 mysqld[218781]: 2020-10-16 12:26:54
139773373455744 [Note] Crash recovery finished.
Oct 16 12:26:54 novi-pfence-01 mysqld[218781]: 2020-10-16 12:26:54
139773373455744 [ERROR] Can't open and lock privilege tables: Table
'mysql.servers' doesn't exist
Oct 16 12:26:54 novi-pfence-01 mysqld[218781]: 2020-10-16 12:26:54
139773373455744 [Note] Server socket created on IP: '10.22.0.82'.
Oct 16 12:26:54 novi-pfence-01 mysqld[218781]: 2020-10-16 12:26:54
139773373455744 [Note] WSREP: disabled, skipping position recovery
Oct 16 12:26:57 novi-pfence-01 packetfence[215534]: pf-mariadb(215534) INFO:
There is a peer with an alive DB. Will attempt to connect to the cluster
(main::startup_clean_
Oct 16 12:26:57 novi-pfence-01 pf-mariadb[215534]: Starting MySQL with command:
mysqld_safe --defaults-file=/usr/local/pf/var/conf/mariadb.cnf
Oct 16 12:26:57 novi-pfence-01 pf-mariadb[215534]: 201016 12:26:57 mysqld_safe
Logging to syslog.
Oct 16 12:26:57 novi-pfence-01 pf-mariadb[215534]: 201016 12:26:57 mysqld_safe
Starting mysqld daemon with databases from /var/lib/mysql
root@novi-pfence-01:/var/lib/mysql# systemctl -l --wait status
packetfence-mariadb
● packetfence-mariadb.service - PacketFence MariaDB instance
Loaded: loaded (/lib/systemd/system/packetfence-mariadb.service; enabled;
vendor preset: enabled)
Active: activating (start) since Fri 2020-10-16 12:26:23 EDT; 35s ago
Process: 215522 ExecStartPre=/usr/local/pf/bin/pfcmd generatemariadbconfig
(code=exited, status=0/SUCCESS)
Main PID: 215534 (pf-mariadb)
Status: "Starting Innodb crash recovery"
Tasks: 23 (limit: 36864)
CGroup: /packetfence.slice/packetfence-base.slice/packetfence-mariadb.service
├─215534 pf-mariadb
├─219013 /bin/bash /usr/bin/mysqld_safe
--defaults-file=/usr/local/pf/var/conf/mariadb.cnf --wsrep-recover
├─219167 /usr/sbin/mysqld
--defaults-file=/usr/local/pf/var/conf/mariadb.cnf --basedir=/usr
--datadir=/var/lib/mysql --plugin-dir=/usr/lib/x86_64-linux-gnu/mari
└─219168 logger -t mysqld -p daemon error
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: Compressed tables use zlib 1.2.8
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: Using Linux native AIO
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: Using SSE crc32 instructions
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: Initializing buffer pool, size = 500.0M
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: Completed initialization of buffer pool
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: Highest supported file format is Barracuda.
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: The log sequence number 1607435 in ibdata file
do not match the log seque
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: Restoring possible half-written data pages from
the doublewrite buffer...
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: 128 rollback segment(s) are active.
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: Waiting for purge to start
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: Percona XtraDB (http://www.percona.com)
5.6.47-87.0 started; log sequenc
root@novi-pfence-01:/var/lib/mysql# systemctl -l --wait status
packetfence-mariadb
● packetfence-mariadb.service - PacketFence MariaDB instance
Loaded: loaded (/lib/systemd/system/packetfence-mariadb.service; enabled;
vendor preset: enabled)
Active: activating (start) since Fri 2020-10-16 12:26:23 EDT; 35s ago
Process: 215522 ExecStartPre=/usr/local/pf/bin/pfcmd generatemariadbconfig
(code=exited, status=0/SUCCESS)
Main PID: 215534 (pf-mariadb)
Status: "Waiting for master thread to exit"
Tasks: 20 (limit: 36864)
CGroup: /packetfence.slice/packetfence-base.slice/packetfence-mariadb.service
├─215534 pf-mariadb
├─219013 /bin/bash /usr/bin/mysqld_safe
--defaults-file=/usr/local/pf/var/conf/mariadb.cnf --wsrep-recover
├─219167 /usr/sbin/mysqld
--defaults-file=/usr/local/pf/var/conf/mariadb.cnf --basedir=/usr
--datadir=/var/lib/mysql --plugin-dir=/usr/lib/x86_64-linux-gnu/mari
└─219168 logger -t mysqld -p daemon error
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] InnoDB: Percona XtraDB (http://www.percona.com)
5.6.47-87.0 started; log sequenc
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Warning] InnoDB: Skipping buffer pool dump/restore during
wsrep recovery.
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] Plugin 'FEEDBACK' is disabled.
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [ERROR] Could not open mysql.plugin table. Some plugins may be
not loaded
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] Recovering after a crash using tc.log
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] Starting crash recovery...
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] Crash recovery finished.
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [ERROR] Can't open and lock privilege tables: Table
'mysql.servers' doesn't exist
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] Server socket created on IP: '10.22.0.82'.
Oct 16 12:26:58 novi-pfence-01 mysqld[219168]: 2020-10-16 12:26:58
139864779369856 [Note] WSREP: disabled, skipping position recovery
The most significant error appears to be that about table ‘mysql.servers’
doesn’t exist, but I don’t know what to do about it.
I can see that after running the command to restart packetfence-mariadb on the
other node, the folder /var/lib/mysql has the following files in it. I think
this means it has contacted the remote server and started the sync, but cannot
be certain.
-rw-rw---- 1 mysql mysql 16384 Oct 16 12:33 aria_log.00000001
-rw-rw---- 1 mysql mysql 52 Oct 16 12:33 aria_log_control
-rw-rw---- 1 mysql mysql 12582912 Oct 16 12:33 ibdata1
-rw-rw---- 1 mysql mysql 50331648 Oct 16 12:33 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 Oct 16 12:28 ib_logfile1
-rw-rw---- 1 mysql mysql 24576 Oct 16 12:33 tc.log
I look forward to any advice available on how to proceed from here.
Thank you,
Jeff
PRIVACY NOTICE: The information contained in this e-mail, including any
attachments, is confidential and intended only for the named recipient(s).
Unauthorized use, disclosure, forwarding, or copying is strictly prohibited and
may be unlawful. If you are not the intended recipient, please delete the
e-mail and any attachments and notify us immediately by return e-mail.
_______________________________________________
PacketFence-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/packetfence-users
