telneting around it appears server1 can reach 8001 on itself but not 2. server 2 can telnet 8001 on server 1 but not itself. I see nothing logged that indicates a conflict tho and both of these besides being on different subnets these servers are virtually identical and firewall free.
CS, cs.temp.m...@gmail.com On Wed, 27 Dec 2023 at 14:38, CS <cs.temp.m...@gmail.com> wrote: > They made it through. With minor changes (ip addresses, library locations, > and logging) the files are accepted and daemons are running but... no dice. > > server1$ tail -f /var/log/kea/kea-dhcp4.log > 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687] > DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file > /var/lib/kea/kea-leases4.csv > 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687] > DHCPSRV_MEMFILE_EXTRACT_EXTENDED_INFO4 extracting extended info saw 0 > leases, extended info sanity checks modified 0 / updated 0 leases and 0 > leases have relay or remote id > 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687] > DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to > 3600 sec > 2023-12-27 22:24:48.486 WARN [kea-dhcp4.dhcpsrv/1495687] > DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic > 2023-12-27 22:24:48.486 INFO [kea-dhcp4.ha-hooks/1495687] > HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the server1 is > in the WAITING state > 2023-12-27 22:24:48.486 INFO [kea-dhcp4.ha-hooks/1495687] > HA_SERVICE_STARTED started high availability service in load-balancing mode > as primary server > 2023-12-27 22:24:48.487 WARN [kea-dhcp4.dhcp4/1495687] > DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size: > 64 > 2023-12-27 22:24:48.487 INFO [kea-dhcp4.dhcp4/1495687] DHCP4_STARTED Kea > DHCPv4 server version 2.4.0 started > 2023-12-27 22:24:58.498 WARN [kea-dhcp4.ha-hooks/1495687] > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 ( > http://xxx:8001): Connection refused > 2023-12-27 22:25:08.510 WARN [kea-dhcp4.ha-hooks/1495687] > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 ( > http://xxx:8001): Connection refused > 2023-12-27 22:25:18.519 WARN [kea-dhcp4.ha-hooks/1495687] > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 ( > http://xxx5:8001): Connection refused > 2023-12-27 22:25:28.531 WARN [kea-dhcp4.ha-hooks/1495687] > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 ( > http://xxx:8001): Connection refused > 2023-12-27 22:25:37.660 INFO [kea-dhcp4.commands/1495687] > COMMAND_RECEIVED Received command 'ha-heartbeat' > 2023-12-27 22:25:38.535 WARN [kea-dhcp4.ha-hooks/1495687] > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 ( > http://xxx:8001): Connection refused > 2023-12-27 22:25:47.674 INFO [kea-dhcp4.commands/1495687] > COMMAND_RECEIVED Received command 'ha-heartbeat' > 2023-12-27 22:25:48.546 WARN [kea-dhcp4.ha-hooks/1495687] > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 ( > http://xxx:8001): Connection refused > 2023-12-27 22:25:48.546 WARN [kea-dhcp4.ha-hooks/1495687] > HA_COMMUNICATION_INTERRUPTED communication with server2 is interrupted > 2023-12-27 22:25:48.546 INFO [kea-dhcp4.ha-hooks/1495687] > HA_STATE_TRANSITION server transitions from WAITING to PARTNER-DOWN state, > partner state is UNDEFINED > 2023-12-27 22:25:48.547 INFO [kea-dhcp4.ha-hooks/1495687] > HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner > while in PARTNER-DOWN state > 2023-12-27 22:25:48.547 INFO [kea-dhcp4.ha-hooks/1495687] > HA_LOCAL_DHCP_ENABLE local DHCP service is enabled while the server1 is in > the PARTNER-DOWN state > 2023-12-27 22:25:57.687 INFO [kea-dhcp4.commands/1495687] > COMMAND_RECEIVED Received command 'ha-heartbeat' > 2023-12-27 22:25:57.690 INFO [kea-dhcp4.commands/1495687] > COMMAND_RECEIVED Received command 'dhcp-disable' > 2023-12-27 22:25:57.691 INFO [kea-dhcp4.commands/1495687] > COMMAND_RECEIVED Received command 'lease4-get-page' > 2023-12-27 22:25:57.693 INFO [kea-dhcp4.commands/1495687] > COMMAND_RECEIVED Received command 'ha-sync-complete-notify' > 2023-12-27 22:25:58.557 WARN [kea-dhcp4.ha-hooks/1495687] > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 ( > http://xxx:8001): Connection refused > 2023-12-27 22:25:58.557 WARN [kea-dhcp4.ha-hooks/1495687] > HA_COMMUNICATION_INTERRUPTED communication with server2 is interrupted > > > server2$ tail -f /var/log/kea/kea-dhcp4.log > 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcp4/1434251] > DHCP4_CONFIG_COMPLETE DHCPv4 server has completed configuration: no IPv4 > subnets!; DDNS: disabled > 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251] > DHCPSRV_MEMFILE_DB opening memory file lease database: type=memfile > universe=4 > 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251] > DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file > /var/lib/kea/kea-leases4.csv.2 > 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251] > DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file > /var/lib/kea/kea-leases4.csv > 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251] > DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to > 3600 sec > 2023-12-27 22:25:26.650 WARN [kea-dhcp4.dhcpsrv/1434251] > DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic > 2023-12-27 22:25:26.650 INFO [kea-dhcp4.ha-hooks/1434251] > HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the server2 is > in the WAITING state > 2023-12-27 22:25:26.650 INFO [kea-dhcp4.ha-hooks/1434251] > HA_SERVICE_STARTED started high availability service in load-balancing mode > as secondary server > 2023-12-27 22:25:26.650 WARN [kea-dhcp4.dhcp4/1434251] > DHCP4_MULTI_THREADING_INFO enabled: no, number of threads: 0, queue size: 0 > 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcp4/1434251] DHCP4_STARTED Kea > DHCPv4 server version 2.2.0 started > 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251] > HA_STATE_TRANSITION server transitions from WAITING to SYNCING state, > partner state is PARTNER-DOWN > 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251] > HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner > while in SYNCING state > 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251] HA_SYNC_START > starting lease database synchronization with server1 > 2023-12-27 22:25:57.693 INFO [kea-dhcp4.ha-hooks/1434251] > HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 0 leases from server1 > 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251] > HA_SYNC_SUCCESSFUL lease database synchronization with server1 completed > successfully in 3.877 ms > 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251] > HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner > state is PARTNER-DOWN > 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251] > HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner > while in READY state > > CS, cs.temp.m...@gmail.com > > > On Wed, 27 Dec 2023 at 11:22, Darren Ankney <darren.ank...@gmail.com> > wrote: > >> Hi, >> >> See attached four files: >> >> ca-server1.json (config for kea-ctrl-agent) >> ca-server2.json (config for kea-ctrl-agent) >> dhcp4-server1.json (config for kea-dhcp4) >> dhcp4-server2.json (config for kea-dhcp4) >> >> These files use port 8000 for kea-ctrl-agent and 8001 for kea-dhcp4 on >> version 2.4.0. They don't do anything other than setup the heartbeats >> for HA (in fact you don't even need to run the control agent as Kea is >> communicating directly). Heartbeats are sent back and forth on port >> 8001 as expected. Can you give these a try and see if they work (in >> testing of course, they won't serve any clients)? I'm genuinely >> curious if they work. Yours should work (unless there is some problem >> with the certificates or something). I didn't notice any reason why >> they wouldn't. >> >> Thank you, >> >> Darren Ankney >> >> PS: I'm not sure if these attachments will make it through to the list. >> >> On Wed, Dec 27, 2023 at 7:16 AM CS <cs.temp.m...@gmail.com> wrote: >> > >> > Kea 2.4.0 >> > >> > On Wed, Dec 27, 2023, 03:18 Darren Ankney <darren.ank...@gmail.com> >> wrote: >> >> >> >> Hi, >> >> >> >> If I may ask, what version of Kea are you using? Some defaults have >> >> changed across versions. >> >> >> >> Thank you, >> >> >> >> Darren Ankney >> >> >> >> On Tue, Dec 26, 2023 at 4:31 PM CS <cs.temp.m...@gmail.com> wrote: >> >> > >> >> > >Please describe what you mean by "it doesn't work". >> >> > I mean I get a pretty useless error: "Unable to connect to Kea >> Control Agent." >> >> > >> >> > > it might be be best to ask Men & Mice about "micetro" and how best >> to set things >> >> > I will at some point, when I find a resource with them. But there >> are two players in this and since kea isn't behaving as expected like you, >> I and the docs said. I'm starting here. >> >> > >> >> > >It actually SHOULDN'T work >> >> > That's my read on it too. But here's proof. The CA config for one >> server. It matches for the other server except certs and ip addresses obv. >> >> > >> >> > "Control-agent": { >> >> > "http-host": "xxx.xx1.xxx.xxx", >> >> > "trust-anchor": "Certificate_Autority.pem", >> >> > "cert-file": "ca1_cert.pem", >> >> > "key-file": "ca1_key.pem", >> >> > "cert-required": true, >> >> > "http-port": 8000, >> >> > "authentication": { >> >> > "type": "basic", >> >> > "realm": "kea-control-agent", >> >> > "clients": [{ >> >> > "user": "baduser", >> >> > "password": "badpassword", >> >> > }] >> >> > }, >> >> > >> >> > And the dhcp4 config, likewise only the small differences between >> the two servers >> >> > >> >> > "hooks-libraries": [{ >> >> > "library": >> "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so", >> >> > "parameters": {} >> >> > },{ >> >> > "library" : >> "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so", >> >> > "parameters": { >> >> > "high-availability": [{ >> >> > "this-server-name": " >> server1.org.org", >> >> > "mode": "load-balancing", >> >> > "heartbeat-delay": 10000, >> >> > "max-response-delay": 60000, >> >> > "max-ack-delay": 5000, >> >> > "max-unacked-clients": 0, >> >> > "require-client-certs": true, >> >> > "trust-anchor": >> "Certificate_Autority.pem", >> >> > "auto-failover": true, >> >> > >> >> > "peers": [{ >> >> > "name": " >> server1.org.org", >> >> > "url": " >> http://xxx.xx1.xxx.xxx:8000/", >> >> > "cert-file": >> "dhcp1_cert.pem", >> >> > "key-file": >> "dhcp1_key.pem", >> >> > "basic-auth-user": >> "baduser", >> >> > >> "basic-auth-password": "badpassword", >> >> > "role": "primary", >> >> > },{ >> >> > "name": " >> server2.org.org", >> >> > "url": " >> http://xxx.xx2.xxx.xxx:8000/", >> >> > "cert-file": >> "dhcp2_cert.pem", >> >> > "key-file": >> "dhcp2_key.pem", >> >> > "role": "secondary", >> >> > "basic-auth-user": >> "baduser", >> >> > >> "basic-auth-password": "badpassword", >> >> > }] >> >> > }] >> >> > >> >> > low and behold it runs. The same nature of daemon status and logs on >> the other server. >> >> > >> >> > $ sudo systemctl restart isc-kea-ctrl-agent.service >> isc-kea-dhcp4-server.service >> >> > $ sudo systemctl status isc-kea-ctrl-agent.service >> isc-kea-dhcp4-server.service >> >> > ● isc-kea-ctrl-agent.service - Kea Control Agent >> >> > Loaded: loaded (/lib/systemd/system/isc-kea-ctrl-agent.service; >> enabled; vendor preset: enabled) >> >> > Active: active (running) since Tue 2023-12-26 20:57:29 UTC; 11s >> ago >> >> > Docs: man:kea-ctrl-agent(8) >> >> > Main PID: 1393724 (kea-ctrl-agent) >> >> > Tasks: 5 (limit: 19052) >> >> > Memory: 2.5M >> >> > CPU: 26ms >> >> > CGroup: /system.slice/isc-kea-ctrl-agent.service >> >> > └─1393724 /usr/sbin/kea-ctrl-agent -c >> /etc/kea/kea-ctrl-agent.conf >> >> > >> >> > Dec 26 20:57:29 kea1 systemd[1]: Started Kea Control Agent. >> >> > >> >> > ● isc-kea-dhcp4-server.service - Kea DHCPv4 Service >> >> > Loaded: loaded >> (/lib/systemd/system/isc-kea-dhcp4-server.service; enabled; vendor preset: >> enabled) >> >> > Active: active (running) since Tue 2023-12-26 20:57:29 UTC; 11s >> ago >> >> > Docs: man:kea-dhcp4(8) >> >> > Main PID: 1393730 (kea-dhcp4) >> >> > Tasks: 9 (limit: 19052) >> >> > Memory: 4.5M >> >> > CPU: 96ms >> >> > CGroup: /system.slice/isc-kea-dhcp4-server.service >> >> > └─1393730 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf >> >> > >> >> > Dec 26 20:57:29 kea1 systemd[1]: isc-kea-dhcp4-server.service: >> Deactivated successfully. >> >> > Dec 26 20:57:29 kea1 systemd[1]: Stopped Kea DHCPv4 Service. >> >> > Dec 26 20:57:29 kea1 systemd[1]: isc-kea-dhcp4-server.service: >> Consumed 1min 28.504s CPU time. >> >> > Dec 26 20:57:29 kea1 systemd[1]: Started Kea DHCPv4 Service. >> >> > >> >> > $ tail -n10 /var/log/kea/kea-ctrl-agent.log >> >> > 2023-12-26 20:59:53.827 INFO [kea-ctrl-agent.ctrl-agent/1393724] >> CTRL_AGENT_COMMAND_RECEIVED command ha-heartbeat received from remote >> address xxx.xxx2.xxx.xxx >> >> > 2023-12-26 20:59:53.828 INFO [kea-ctrl-agent.ctrl-agent/1393724] >> CTRL_AGENT_COMMAND_FORWARDED command ha-heartbeat successfully forwarded to >> the service dhcp4 from remote address xxx.xx2.xxx.xxx >> >> > 2023-12-26 21:00:03.843 INFO [kea-ctrl-agent.auth/1393724] >> HTTP_CLIENT_REQUEST_AUTHORIZED received HTTP request authorized for >> 'baduser' >> >> > 2023-12-26 21:00:03.843 INFO [kea-ctrl-agent.commands/1393724] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 21:00:03.843 INFO [kea-ctrl-agent.ctrl-agent/1393724] >> CTRL_AGENT_COMMAND_RECEIVED command ha-heartbeat received from remote >> address xxx.xxx2.xxx.xxx >> >> > 2023-12-26 21:00:03.844 INFO [kea-ctrl-agent.ctrl-agent/1393724] >> CTRL_AGENT_COMMAND_FORWARDED command ha-heartbeat successfully forwarded to >> the service dhcp4 from remote address xxx.xxx2.xxx.xxx >> >> > 2023-12-26 21:00:13.859 INFO [kea-ctrl-agent.auth/1393724] >> HTTP_CLIENT_REQUEST_AUTHORIZED received HTTP request authorized for >> 'baduser' >> >> > 2023-12-26 21:00:13.859 INFO [kea-ctrl-agent.commands/1393724] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 21:00:13.859 INFO [kea-ctrl-agent.ctrl-agent/1393724] >> CTRL_AGENT_COMMAND_RECEIVED command ha-heartbeat received from remote >> address xxx.xxx2.xxx.xxx >> >> > 2023-12-26 21:00:13.860 INFO [kea-ctrl-agent.ctrl-agent/1393724] >> CTRL_AGENT_COMMAND_FORWARDED command ha-heartbeat successfully forwarded to >> the service dhcp4 from remote address xxx.xxx2.xxx.xxx >> >> > $ tail -n10 /var/log/kea/kea-dhcp4.log >> >> > 2023-12-26 20:58:53.728 INFO [kea-dhcp4.commands/1393730] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 20:59:03.745 INFO [kea-dhcp4.commands/1393730] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 20:59:13.762 INFO [kea-dhcp4.commands/1393730] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 20:59:23.777 INFO [kea-dhcp4.commands/1393730] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 20:59:33.793 INFO [kea-dhcp4.commands/1393730] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 20:59:43.811 INFO [kea-dhcp4.commands/1393730] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 20:59:53.827 INFO [kea-dhcp4.commands/1393730] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 21:00:03.844 INFO [kea-dhcp4.commands/1393730] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 21:00:13.859 INFO [kea-dhcp4.commands/1393730] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > 2023-12-26 21:00:23.875 INFO [kea-dhcp4.commands/1393730] >> COMMAND_RECEIVED Received command 'ha-heartbeat' >> >> > >> >> > And changing the CA or the server HA paramersts to port 8001 without >> changing the other (and the other server results in "connection refused" >> logs. It obv wants the CA port to match the HA parameters port despite what >> we and the documentation suggests... >> >> > >> >> > CS, cs.temp.m...@gmail.com >> >> > >> >> > >> >> > On Mon, 25 Dec 2023 at 02:45, Darren Ankney <darren.ank...@gmail.com> >> wrote: >> >> >> >> >> >> Hi, >> >> >> >> >> >> It actually SHOULDN'T work to set your control agent and >> >> >> multi-threaded HA listener to the same port as only one of the >> >> >> applications should be able to setup a listener on that port. >> Please >> >> >> describe what you mean by "it doesn't work". I'm thinking it might >> be >> >> >> be best to ask Men & Mice about "micetro" and how best to set things >> >> >> up there. >> >> >> >> >> >> Thank you, >> >> >> >> >> >> Darren Ankney >> >> >> >> >> >> On Thu, Dec 21, 2023 at 6:47 PM CS <cs.temp.m...@gmail.com> wrote: >> >> >> > >> >> >> > Hi all, >> >> >> > Moving on from my failure to start and logging issues (thank you >> for your help btw!) I now don't have my heartbeat/control_agent working >> correctly. >> >> >> > >> >> >> > It works fine so long as I set the ports of my control agents and >> ha hook parameters to be the same (IE 8000 or 8001) >> >> >> > >> >> >> > However I am unable to tie the tiny cluster into micetro, >> probably because the CA port is occupied with HA heartbeats? >> >> >> > >> >> >> > Looking to these examples: >> >> >> > >> https://github.com/isc-projects/kea/tree/master/doc/examples/template-ha-mt-tls >> >> >> > >> >> >> > Documentation points out >> >> >> > //This specifies the port CA will listen on. >> >> >> > // If enabling HA and multi-threading, the 8000 port is >> used by the HA >> >> >> > // hook library http listener. When using HA hook library >> with >> >> >> > // multi-threading to function, make sure the port used >> by dedicated >> >> >> > // listener is different (e.g. 8001) than the one used by >> CA. Note >> >> >> > // the commands should still be sent via CA. The >> dedicated listener >> >> >> > // is specifically for HA updates only. >> >> >> > >> >> >> > However, how to have a dedicated port for HA and a different one >> for CA escapes me. >> >> >> > >> >> >> > CS, cs.temp.m...@gmail.com >> >> >> > -- >> >> >> > ISC funds the development of this software with paid support >> subscriptions. Contact us at https://www.isc.org/contact/ for more >> information. >> >> >> > >> >> >> > To unsubscribe visit >> https://lists.isc.org/mailman/listinfo/kea-users. >> >> >> > >> >> >> > Kea-users mailing list >> >> >> > Kea-users@lists.isc.org >> >> >> > https://lists.isc.org/mailman/listinfo/kea-users >> >> >> -- >> >> >> ISC funds the development of this software with paid support >> subscriptions. Contact us at https://www.isc.org/contact/ for more >> information. >> >> >> >> >> >> To unsubscribe visit >> https://lists.isc.org/mailman/listinfo/kea-users. >> >> >> >> >> >> Kea-users mailing list >> >> >> Kea-users@lists.isc.org >> >> >> https://lists.isc.org/mailman/listinfo/kea-users >> >
-- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users