Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 03.07.23 um 13:53 schrieb Stefan G. Weichinger:

I *think* communication between stork and kea is encrypted now, just not 
fully verified.


As far as I understand I have mode "Encryption" now as described here:

https://kea.readthedocs.io/en/kea-2.2.0/arm/security.html#tls-https-support

... but not "mutual authentication".


--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 30.06.23 um 18:23 schrieb Stefan G. Weichinger:


So it's very likely that adding that IP SAN to the cert fixes things.


Replaced the CA and the certs/keys in kea-dhcp4.conf and 
kea-ctrl-agent.conf.


That changes things, but doesn't fully fix my issues.

Should the cert in kea-ctrl-agent.conf be another one or the same one? 
Sorry for my confusion.


If I turn on

"cert-required" in the ctrl-agent and 
STORK_AGENT_SKIP_TLS_CERT_VERIFICATION=false for the stork-agent things 
fail.


No more problems with a missing IP SAN, but still verification errors.

The ca.crt in /etc/kea is the same on both machines.

And I added that ca.crt to the system's key storage 
("update-ca-certificates").


I *think* communication between stork and kea is encrypted now, just not 
fully verified.


--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Eric Graham]

Stefan,

Sounds good! That should be all... I'd just test by first trying a simple cURL 
to the control agent, maybe. It shouldn't complain about an unknown cert; if it 
does, then there's clearly an issue. I also ran into issues where, for some 
unknown reason, my certificates weren't validating against the CA... so 
checking that is maybe worth it, if the cURL test fails. But if that was the 
case, I'd expect heartbeats between the two servers to be failing as well.

Good luck to you with your changes. 

Eric Graham
DevOps Specialist
Direct: 605.990.1859
eric.gra...@vantagepnt.com<mailto:eric.gra...@vantagepnt.com>
[cid:644cffd4-1051-44e1-91fc-5e4271301454]

From: Stefan G. Weichinger 
Sent: Friday, June 30, 2023 11:23 AM
To: Eric Graham ; kea-users@lists.isc.org 

Subject: Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork 
and dhcp4 gets lost

CAUTION: This email originated outside the organization. Do not click any links 
or attachments unless you have verified the sender.

Am 30.06.23 um 17:53 schrieb Eric Graham:
> Stefan,
>
> I've been down this road and the short answer is to not bother trying to
> use the various options to skip certificate verification. Those settings
> don't do what you (I) think they do, and it's easier to just make the
> certs work.
>
> When you generate the certificates under your CA, add the IP address of
> each server as an IP SAN. For example, given a key, CA, and CSR, this is
> how I make a certificate:
>
> *HOSTNAME='1.2.3.4'*
> *openssl x509 -req -sha512 -days 365 -in ${HOSTNAME}.csr -CA ca.crt
> -CAkey cakey.pem -CAcreateserial -out ${HOSTNAME}.crt -extensions SAN
> -extfile <(printf "[SAN]\nsubjectAltName=IP:${HOSTNAME}")*
> *
> *
> In my case, I only care to make the certificate work for IP address, so
> you'll need to adjust the various options (obviously). When you're done,
> use the *-print* option to openssl on *${HOSTNAME}.crt* to double check
> that the SAN is added.
>
> Then, double-double check that the CA is imported on both Kea servers,
> the Stork server, and since you mentioned Docker - also inside any
> containerized version of the aforementioned.
>
> Again, I don't change any of the verification settings, nor any of the
> certificates except the ones that I created for Kea to use. Hope this helps.

Thanks for pointing this out. My certs were done like:

openssl req -nodes -newkey rsa:2048 -keyout server_adc1.key -out
server_adc1.csr -subj
"/C=some/ST=some/L=there/O=ISC-Kea/OU=adc1/CN=adc1/emailAddress=a...@my.tld"

so I have no IP SAN, right (grepped that command from a gist on github
and modified it).

I wanted to get it right with FQDNs in there etc ...

I will give your approach a try next week or so, currently on the train
and not touching anything anymore today.

How to double-double-check the CA import? I added it to
/usr/local/share/ca-certificates/ and ran update-ca-certificates , so
the ca.crt should be in the system's keystore.

Is that enough to make stork trust it?

I assume so as I didn't find a specific setting/variable to define a TLS
CA for the stork-agent.

So it's very likely that adding that IP SAN to the cert fixes things.

I will see next week ;-)

Thanks, have a nice weekend.
Stefan
-- 
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 30.06.23 um 17:53 schrieb Eric Graham:

Stefan,

I've been down this road and the short answer is to not bother trying to 
use the various options to skip certificate verification. Those settings 
don't do what you (I) think they do, and it's easier to just make the 
certs work.


When you generate the certificates under your CA, add the IP address of 
each server as an IP SAN. For example, given a key, CA, and CSR, this is 
how I make a certificate:


*HOSTNAME='1.2.3.4'*
*openssl x509 -req -sha512 -days 365 -in ${HOSTNAME}.csr -CA ca.crt 
-CAkey cakey.pem -CAcreateserial -out ${HOSTNAME}.crt -extensions SAN 
-extfile <(printf "[SAN]\nsubjectAltName=IP:${HOSTNAME}")*

*
*
In my case, I only care to make the certificate work for IP address, so 
you'll need to adjust the various options (obviously). When you're done, 
use the *-print* option to openssl on *${HOSTNAME}.crt* to double check 
that the SAN is added.


Then, double-double check that the CA is imported on both Kea servers, 
the Stork server, and since you mentioned Docker - also inside any 
containerized version of the aforementioned.


Again, I don't change any of the verification settings, nor any of the 
certificates except the ones that I created for Kea to use. Hope this helps.


Thanks for pointing this out. My certs were done like:

openssl req -nodes -newkey rsa:2048 -keyout server_adc1.key -out 
server_adc1.csr -subj 
"/C=some/ST=some/L=there/O=ISC-Kea/OU=adc1/CN=adc1/emailAddress=a...@my.tld"


so I have no IP SAN, right (grepped that command from a gist on github 
and modified it).


I wanted to get it right with FQDNs in there etc ...

I will give your approach a try next week or so, currently on the train 
and not touching anything anymore today.


How to double-double-check the CA import? I added it to 
/usr/local/share/ca-certificates/ and ran update-ca-certificates , so 
the ca.crt should be in the system's keystore.


Is that enough to make stork trust it?

I assume so as I didn't find a specific setting/variable to define a TLS 
CA for the stork-agent.


So it's very likely that adding that IP SAN to the cert fixes things.

I will see next week ;-)

Thanks, have a nice weekend.
Stefan
--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Eric Graham]

Stefan,

I've been down this road and the short answer is to not bother trying to use 
the various options to skip certificate verification. Those settings don't do 
what you (I) think they do, and it's easier to just make the certs work.

When you generate the certificates under your CA, add the IP address of each 
server as an IP SAN. For example, given a key, CA, and CSR, this is how I make 
a certificate:

HOSTNAME='1.2.3.4'
openssl x509 -req -sha512 -days 365 -in ${HOSTNAME}.csr -CA ca.crt -CAkey 
cakey.pem -CAcreateserial -out ${HOSTNAME}.crt -extensions SAN -extfile 
<(printf "[SAN]\nsubjectAltName=IP:${HOSTNAME}")

In my case, I only care to make the certificate work for IP address, so you'll 
need to adjust the various options (obviously). When you're done, use the 
-print option to openssl on ${HOSTNAME}.crt to double check that the SAN is 
added.

Then, double-double check that the CA is imported on both Kea servers, the 
Stork server, and since you mentioned Docker - also inside any containerized 
version of the aforementioned.

Again, I don't change any of the verification settings, nor any of the 
certificates except the ones that I created for Kea to use. Hope this helps.

Eric Graham
DevOps Specialist
Direct: 605.990.1859
eric.gra...@vantagepnt.com<mailto:eric.gra...@vantagepnt.com>
[cid:5d14fe52-b13e-4292-9ba3-9e7c1ad07c1c]

From: Kea-users  on behalf of Stefan G. 
Weichinger 
Sent: Friday, June 30, 2023 6:13 AM
To: kea-users@lists.isc.org 
Subject: Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork 
and dhcp4 gets lost

CAUTION: This email originated outside the organization. Do not click any links 
or attachments unless you have verified the sender.

Am 30.06.23 um 12:16 schrieb Stefan G. Weichinger:
>
> After some more restarting and re-registering currently stork looks good.
>
> I assume currently the stork-agents talk to the kea-ctrl-agents
> unencrypted ... I am not 100% sure yet.

Tested flipping this:

kea-ctrl-agent.conf:"cert-required": false

to true

When doing this, the stork-agent has issues trusting the cert:

Jun 30 13:07:30 adc1 stork-agent[759628]: time="2023-06-30 13:07:30"
level="error" msg="Problem fetching stats from Kea: Post
\"https://10.0.0.231:8000/\": remote error: tls: unknown certificate
authority\nproblem sending POST to
https://10.0.0.231:8000/\nisc.org/stork/agent.(*HTTPClient).Call\n\t/builds/isc-projects/stork/backend/agent/caclient.go:105\nisc.org/stork/agent.(*PromKeaExporter).sendCommandToKeaCA\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:876\nisc.org/stork/agent.(*PromKeaExporter).collectStats\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:838\nisc.org/stork/agent.(*PromKeaExporter).statsCollectorLoop\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:710\nruntime.goexit\n\t/builds/isc-projects/stork/tools/golang/go/src/runtime/asm_amd64.s:1594\nproblem
getting stats from
Kea\nisc.org/stork/agent.(*PromKeaExporter).sendCommandToKeaCA\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:878\nisc.org/stork/agent.(*PromKeaExporter).collectStats\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:838\nisc.org/stork/agent.(*PromKeaExporter).statsCollectorLoop\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:710\nruntime.goexit\n\t/builds/isc-projects/stork/tools/golang/go/src/runtime/asm_amd64.s:1594"
file="  promkeaexporter.go:841  "
Jun 30 13:07:30 adc1 stork-agent[759628]: time="2023-06-30 13:07:30"
level="error" msg="Some errors were encountered while collecting stats
from Kea: Post \"https://10.0.0.231:8000/\": remote error: tls: unknown
certificate authority\nproblem sending POST to
https://10.0.0.231:8000/\nisc.org/stork/agent.(*HTTPClient).Call\n\t/builds/isc-projects/stork/backend/agent/caclient.go:105\nisc.org/stork/agent.(*PromKeaExporter).sendCommandToKeaCA\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:876\nisc.org/stork/agent.(*PromKeaExporter).collectStats\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:838\nisc.org/stork/agent.(*PromKeaExporter).statsCollectorLoop\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:710\nruntime.goexit\n\t/builds/isc-projects/stork/tools/golang/go/src/runtime/asm_amd64.s:1594\nproblem
getting stats from
Kea\nisc.org/stork/agent.(*PromKeaExporter).sendCommandToKeaCA\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:878\nisc.org/stork/agent.(*PromKeaExporter).collectStats\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:838\nisc.org/stork/agent.(*PromKeaExporter).statsCollectorLoop\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:710\nruntime.goexit\n\t/builds/isc-projects/stork/tools/golang/go/src/runtime/asm_amd64.s:1594"
file="  promkeaexporter.go:712  "

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 30.06.23 um 12:16 schrieb Stefan G. Weichinger:


After some more restarting and re-registering currently stork looks good.

I assume currently the stork-agents talk to the kea-ctrl-agents 
unencrypted ... I am not 100% sure yet.


Tested flipping this:

kea-ctrl-agent.conf:"cert-required": false

to true

When doing this, the stork-agent has issues trusting the cert:

Jun 30 13:07:30 adc1 stork-agent[759628]: time="2023-06-30 13:07:30" 
level="error" msg="Problem fetching stats from Kea: Post 
\"https://10.0.0.231:8000/\": remote error: tls: unknown certificate 
authority\nproblem sending POST to 
https://10.0.0.231:8000/\nisc.org/stork/agent.(*HTTPClient).Call\n\t/builds/isc-projects/stork/backend/agent/caclient.go:105\nisc.org/stork/agent.(*PromKeaExporter).sendCommandToKeaCA\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:876\nisc.org/stork/agent.(*PromKeaExporter).collectStats\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:838\nisc.org/stork/agent.(*PromKeaExporter).statsCollectorLoop\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:710\nruntime.goexit\n\t/builds/isc-projects/stork/tools/golang/go/src/runtime/asm_amd64.s:1594\nproblem 
getting stats from 
Kea\nisc.org/stork/agent.(*PromKeaExporter).sendCommandToKeaCA\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:878\nisc.org/stork/agent.(*PromKeaExporter).collectStats\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:838\nisc.org/stork/agent.(*PromKeaExporter).statsCollectorLoop\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:710\nruntime.goexit\n\t/builds/isc-projects/stork/tools/golang/go/src/runtime/asm_amd64.s:1594" 
file="  promkeaexporter.go:841  "
Jun 30 13:07:30 adc1 stork-agent[759628]: time="2023-06-30 13:07:30" 
level="error" msg="Some errors were encountered while collecting stats 
from Kea: Post \"https://10.0.0.231:8000/\": remote error: tls: unknown 
certificate authority\nproblem sending POST to 
https://10.0.0.231:8000/\nisc.org/stork/agent.(*HTTPClient).Call\n\t/builds/isc-projects/stork/backend/agent/caclient.go:105\nisc.org/stork/agent.(*PromKeaExporter).sendCommandToKeaCA\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:876\nisc.org/stork/agent.(*PromKeaExporter).collectStats\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:838\nisc.org/stork/agent.(*PromKeaExporter).statsCollectorLoop\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:710\nruntime.goexit\n\t/builds/isc-projects/stork/tools/golang/go/src/runtime/asm_amd64.s:1594\nproblem 
getting stats from 
Kea\nisc.org/stork/agent.(*PromKeaExporter).sendCommandToKeaCA\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:878\nisc.org/stork/agent.(*PromKeaExporter).collectStats\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:838\nisc.org/stork/agent.(*PromKeaExporter).statsCollectorLoop\n\t/builds/isc-projects/stork/backend/agent/promkeaexporter.go:710\nruntime.goexit\n\t/builds/isc-projects/stork/tools/golang/go/src/runtime/asm_amd64.s:1594" 
file="  promkeaexporter.go:712  "
Jun 30 13:07:34 adc1 kea-ctrl-agent[759731]: INFO 
HTTP_CONNECTION_HANDSHAKE_FAILED TLS handshake with 10.0.0.231 failed 
with certificate verify failed



And this while the agent.env has:

STORK_AGENT_SKIP_TLS_CERT_VERIFICATION=true

So I have to figure out how to make the stork-agent trust that cert.

Do I have to modify /var/lib/stork-agent/certs/ca.pem?

As far as I understand the files there are generated while registering 
the stork-agent.


thanks for any help, I think I am close to getting this right

--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

After some more restarting and re-registering currently stork looks good.

I assume currently the stork-agents talk to the kea-ctrl-agents 
unencrypted ... I am not 100% sure yet.



--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Jun 30 11:37:22 adc1 stork-agent[754889]: time="2023-06-30 11:37:22" 
level="info" msg="   kea: control: https://10.0.0.231:8000/ (auth key: 
not found)"


Jun 30 11:37:43 adc1 stork-agent[754889]: time="2023-06-30 11:37:43" 
level="info" msg="Loaded server cert: 
/var/lib/stork-agent/certs/cert.pem and key: 
/var/lib/stork-agent/certs/key.pem\n" file="agent.go:104  "
Jun 30 11:37:43 adc1 stork-agent[754889]: time="2023-06-30 11:37:43" 
level="info" msg="Loaded CA cert: /var/lib/stork-agent/certs/ca.pem\n" 
file="agent.go:78   "
Jun 30 11:37:43 adc1 stork-agent[754889]: time="2023-06-30 11:37:43" 
level="error" msg="Failed to forward commands to Kea CA: Post 
\"http://10.0.0.231:8000/\": EOF\nproblem sending POST to 
http://10.0.0.231:8000/\nisc.org/stork/agent.(*HTTPClient).Call\n\t/builds/isc-projects/stork/backend/agent/caclient.go:105\nisc.org/stork/agent.(*StorkAgent).ForwardToKeaOverHTTP\n\t/builds/isc-projects/stork/backend/agent/agent.go:334\nisc.org/stork/api._Agent_ForwardToKeaOverHTTP_Handler\n\t/builds/isc-projects/stork/backend/api/agent_grpc.pb.go:244\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/builds/isc-projects/stork/tools/golang/gopath/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1336\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/builds/isc-projects/stork/tools/golang/gopath/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1704\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/builds/isc-projects/stork/tools/golang/gopath/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:965\nruntime.goexit\n\t/builds/isc-projects/stork/tools/golang/go/src/runtime/asm_amd64.s:1594" 
file="agent.go:338  " URL="http://10.0.0.231:8000/;




I re-registered the stork-agents successfully.

Communication between stork-agent and kea-ctrl-agent is broken.

Sure, I read and follow docs etc ...

--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 29.06.23 um 16:56 schrieb Eric Graham:
My deployments have a single CA that's used as trust-anchor on both 
machines, and then the certificates are signed by the CA. The CA is 
further added to the systems' trust stores. I haven't tried what you're 
doing (sorry).


I decided to set up a small local CA and generate 2 server certs with it.

Imported the CA to the debian keystore as well.

I have a working communication now between the kea-dhcp4-daemons using 
these TLS-certs: great.


-> the DHCP-cluster works


But my setup with the stork-agents and the stork-server is broken now.

The agents seem to contact the kea-control-agents by IP, so the certs 
(which don't contain IP SANs according to the ugly error messages) don't 
match.


* Do I have to put anything into: /var/lib/stork-agent/certs ?

* I set (in /etc/stork/agent.env):

STORK_AGENT_SKIP_TLS_CERT_VERIFICATION=true

doesn't help.

* I tried in /etc/kea/kea-ctrl-agent.conf:

"cert-required": false

A bit lost currently, sorry.



--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 29.06.23 um 16:56 schrieb Eric Graham:
My deployments have a single CA that's used as trust-anchor on both 
machines, and then the certificates are signed by the CA. The CA is 
further added to the systems' trust stores. I haven't tried what you're 
doing (sorry).


That's something I hear all the time ;-)

I wonder if I could safely change just one of the 2 servers to keep up 
the cluster while introducing TLS.


Testing in a docker-stack leads to issues with DNS and hostnames, it's 
not the same.

--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Eric Graham]

My deployments have a single CA that's used as trust-anchor on both machines, 
and then the certificates are signed by the CA. The CA is further added to the 
systems' trust stores. I haven't tried what you're doing (sorry).

Eric Graham
DevOps Specialist
Direct: 605.990.1859
eric.gra...@vantagepnt.com<mailto:eric.gra...@vantagepnt.com>
[cid:ff23e65c-e61f-497c-bea5-91f13f197392]

From: Stefan G. Weichinger 
Sent: Thursday, June 29, 2023 9:04 AM
To: Eric Graham ; kea-users@lists.isc.org 

Subject: Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork 
and dhcp4 gets lost

CAUTION: This email originated outside the organization. Do not click any links 
or attachments unless you have verified the sender.

Am 29.06.23 um 15:34 schrieb Eric Graham:
> Stefan,
>
> I think so, but I'm not sure if it's best practice to share that
> certificate with Kea since you'd need to open up permissions a little
> and allow Kea to read the private key. If you have no qualms with that
> note, then it's probably worth an attempt, at least. Since Kea shouldn't
> be running as root, you may need to change group ownership of the certs
> or use fACLs.

I could copy them over to /var/lib/kea and adjust things.

Prepared that already

As far as I understand the CAs have to be placed "cross-wise":

server1 has to use ca_server2.pem as trust-anchor


server2 has to use ca_server1.pem as trust-anchor

Right?

I haven't started editing things yet, can't risk downtime while people
are working there.


-- 
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 29.06.23 um 15:34 schrieb Eric Graham:

Stefan,

I think so, but I'm not sure if it's best practice to share that 
certificate with Kea since you'd need to open up permissions a little 
and allow Kea to read the private key. If you have no qualms with that 
note, then it's probably worth an attempt, at least. Since Kea shouldn't 
be running as root, you may need to change group ownership of the certs 
or use fACLs.


I could copy them over to /var/lib/kea and adjust things.

Prepared that already

As far as I understand the CAs have to be placed "cross-wise":

server1 has to use ca_server2.pem as trust-anchor


server2 has to use ca_server1.pem as trust-anchor

Right?

I haven't started editing things yet, can't risk downtime while people 
are working there.



--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Eric Graham]

Stefan,

I think so, but I'm not sure if it's best practice to share that certificate 
with Kea since you'd need to open up permissions a little and allow Kea to read 
the private key. If you have no qualms with that note, then it's probably worth 
an attempt, at least. Since Kea shouldn't be running as root, you may need to 
change group ownership of the certs or use fACLs.

Eric Graham
DevOps Specialist
Direct: 605.990.1859
eric.gra...@vantagepnt.com<mailto:eric.gra...@vantagepnt.com>
[cid:5b21c730-a772-4f63-a022-cd498fb2bc5e]

From: Kea-users  on behalf of Stefan G. 
Weichinger 
Sent: Thursday, June 29, 2023 3:02 AM
To: kea-users@lists.isc.org 
Subject: Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork 
and dhcp4 gets lost

CAUTION: This email originated outside the organization. Do not click any links 
or attachments unless you have verified the sender.

Am 28.06.23 um 09:28 schrieb Stefan G. Weichinger:
> Am 27.06.23 um 17:17 schrieb Eric Graham:
>> Stefan,
>>
>> Make sure that when you change the password, you also change it in
>> Stork and in the HA hook config on each daemon of each server.
>>
>> I am not aware of documentation from ISC for generating certificates,
>
> this:
>
> https://github.com/isc-projects/kea/blob/master/src/lib/asiolink/testutils/ca/doc.txt
>
> ?

The two Kea-Servers are also samba-AD-DCs : so they have their own
AD-related TLS-certs here:

# ls -l /var/lib/samba/private/tls
insgesamt 12
-rw-r--r-- 1 root root 2074 30. Nov 2022  ca.pem
-rw-r--r-- 1 root root 2078 30. Nov 2022  cert.pem
-rw--- 1 root root 3243 30. Nov 2022  key.pem

May I "simply" use these for kea as well? I assume so ...
--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users
-- 
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 28.06.23 um 09:28 schrieb Stefan G. Weichinger:

Am 27.06.23 um 17:17 schrieb Eric Graham:

Stefan,

Make sure that when you change the password, you also change it in 
Stork and in the HA hook config on each daemon of each server.


I am not aware of documentation from ISC for generating certificates, 


this:

https://github.com/isc-projects/kea/blob/master/src/lib/asiolink/testutils/ca/doc.txt

?


The two Kea-Servers are also samba-AD-DCs : so they have their own 
AD-related TLS-certs here:


# ls -l /var/lib/samba/private/tls
insgesamt 12
-rw-r--r-- 1 root root 2074 30. Nov 2022  ca.pem
-rw-r--r-- 1 root root 2078 30. Nov 2022  cert.pem
-rw--- 1 root root 3243 30. Nov 2022  key.pem

May I "simply" use these for kea as well? I assume so ...
--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 27.06.23 um 17:17 schrieb Eric Graham:

Stefan,

Make sure that when you change the password, you also change it in Stork 
and in the HA hook config on each daemon of each server.


I am not aware of documentation from ISC for generating certificates, 


this:

https://github.com/isc-projects/kea/blob/master/src/lib/asiolink/testutils/ca/doc.txt

?

--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 27.06.23 um 17:17 schrieb Eric Graham:

Stefan,

Make sure that when you change the password, you also change it in Stork 
and in the HA hook config on each daemon of each server.


I have done that and cross-checked things but somehow communication 
wasn't established again. Had to roll back to get it working again asap.


Would be better to do that in the evening ... when I am tired ;-)

I am not aware of documentation from ISC for generating certificates, 
but here is an article I found that should get you started: 
https://node-security.com/posts/openssl-creating-a-ca/ 

You will want to make a CA in this case, and not just self-signed 
certificates. Make sure if you make the certificates for IP and not 
hostname, that you add the IP to the SAN field of the certificates. 
Here's an article from Red Hat about trusting the CA on each host: 
https://www.redhat.com/sysadmin/ca-certificates-cli 
 Even if you're not 
in Red Hat-land, it'll get you started.


Thanks for the pointers.

Did such cert-generation quite a few times already (back then before 
LetsEncrypt) but hoped for some helpful script or so.


I'll try this in the next days.

--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Eric Graham]

Stefan,

Make sure that when you change the password, you also change it in Stork and in 
the HA hook config on each daemon of each server.

I am not aware of documentation from ISC for generating certificates, but here 
is an article I found that should get you started: 
https://node-security.com/posts/openssl-creating-a-ca/
You will want to make a CA in this case, and not just self-signed certificates. 
Make sure if you make the certificates for IP and not hostname, that you add 
the IP to the SAN field of the certificates. Here's an article from Red Hat 
about trusting the CA on each host: 
https://www.redhat.com/sysadmin/ca-certificates-cli Even if you're not in Red 
Hat-land, it'll get you started.


Eric Graham
DevOps Specialist
Direct: 605.990.1859
eric.gra...@vantagepnt.com<mailto:eric.gra...@vantagepnt.com>
[cid:2e6a83d9-9c45-41bc-86d4-fdd91ce6a9f1]

From: Stefan G. Weichinger 
Sent: Tuesday, June 27, 2023 2:57 AM
To: Eric Graham ; kea-users@lists.isc.org 

Cc: Darren Ankney 
Subject: Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork 
and dhcp4 gets lost

CAUTION: This email originated outside the organization. Do not click any links 
or attachments unless you have verified the sender.

Am 23.06.23 um 17:34 schrieb Eric Graham:
> Stefan,
>
> Please be aware that you posted a password in your control agent config.
> I strongly recommend replacing it.
>
> You may prefer to put the socket in /var. Cleaning of /tmp is
> distro-dependent behavior. You'll need to make that change (to the
> socket path) in the control agent and DHCP configs on both servers.
> Stork will pick up the change automatically (without any config
> changes), but the agent may need a restart, as well as all Kea services.

Changed the socket path, we'll see if that improves stability.

Changing the password didn't work yet, I had to roll back. I'll try that
again later.

I have basic-auth in place, but no TLS enabled yet. This might be the
time to add this also, although the 2 machines run in a rather protected
environment. It's just better, and state of the art, to use TLS ...

Any pointers to the kea-docs how to generate working certs? I assume
they could be rather dummy style ...

thanks, regards, Stefan


-- 
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 23.06.23 um 17:34 schrieb Eric Graham:

Stefan,

Please be aware that you posted a password in your control agent config. 
I strongly recommend replacing it.


You may prefer to put the socket in /var. Cleaning of /tmp is 
distro-dependent behavior. You'll need to make that change (to the 
socket path) in the control agent and DHCP configs on both servers. 
Stork will pick up the change automatically (without any config 
changes), but the agent may need a restart, as well as all Kea services.


Changed the socket path, we'll see if that improves stability.

Changing the password didn't work yet, I had to roll back. I'll try that 
again later.


I have basic-auth in place, but no TLS enabled yet. This might be the 
time to add this also, although the 2 machines run in a rather protected 
environment. It's just better, and state of the art, to use TLS ...


Any pointers to the kea-docs how to generate working certs? I assume 
they could be rather dummy style ...


thanks, regards, Stefan


--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 23.06.23 um 17:34 schrieb Eric Graham:

Stefan,

Please be aware that you posted a password in your control agent config. 
I strongly recommend replacing it.


Yes, thanks, I noticed it after sending the mail.

You may prefer to put the socket in /var. Cleaning of /tmp is 
distro-dependent behavior. You'll need to make that change (to the 
socket path) in the control agent and DHCP configs on both servers. 
Stork will pick up the change automatically (without any config 
changes), but the agent may need a restart, as well as all Kea services.


Yes again. I *think* I took this path out of the Kea-ARM, is that 
possible? I will adjust that asap.


What do you suggest? I'd put it into "/var/lib/kea" ...

--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Eric Graham]

Stefan,

Please be aware that you posted a password in your control agent config. I 
strongly recommend replacing it.

You may prefer to put the socket in /var. Cleaning of /tmp is distro-dependent 
behavior. You'll need to make that change (to the socket path) in the control 
agent and DHCP configs on both servers. Stork will pick up the change 
automatically (without any config changes), but the agent may need a restart, 
as well as all Kea services.

Eric Graham
DevOps Specialist
Direct: 605.990.1859
eric.gra...@vantagepnt.com<mailto:eric.gra...@vantagepnt.com>
[cid:17380953-0d73-4ca6-97ae-ed624e23c4b6]

From: Kea-users  on behalf of Darren Ankney 

Sent: Friday, June 23, 2023 6:01 AM
To: kea-users@lists.isc.org 
Subject: Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork 
and dhcp4 gets lost

CAUTION: This email originated outside the organization. Do not click any links 
or attachments unless you have verified the sender.

Hi Stefan,

I imagine your socket file is being removed during routine OS cleaning
of /tmp/  I would not store it there in production (I do often do that
in testing, however).  If the socket file gets removed, Kea will not
recreate it until a restart.

Thank you,

Darren Ankney

On Fri, Jun 23, 2023 at 4:21 AM Stefan G. Weichinger  wrote:
>
> Am 22.06.23 um 19:35 schrieb Eric Graham:
> > Check the permissions of the control socket on your server:
> >
> > * Confirm that the Kea control agent can read it * Confirm that
> > SELinux is not denying the attempt
>
> Eric, thanks for your help.
>
> If it works most of the time ... why wouldn't the permissions be OK then?
>
> OK:
>
> root@adc1:/etc/kea# ls -l /tmp/kea4*
> srwxr-xr-x 1 _kea _kea 0 22. Jun 07:35 /tmp/kea4-ctrl-socket
> -rw--- 1 _kea _kea 0 22. Jun 07:35 /tmp/kea4-ctrl-socket.lock
>
>
> root@adc2:/etc/kea# ls -l /tmp/kea4*
> srwxr-xr-x 1 _kea _kea 0 17. Mai 08:18 /tmp/kea4-ctrl-socket
> -rw--- 1 _kea _kea 0 17. Mai 08:18 /tmp/kea4-ctrl-socket.lock
>
> SElinux: don't know how to check that right now ...
>
>
> > Your control agent on the other server isn't configured to connect to
> >  the socket for dhcp4 on your primary server, correct?
>
> As far as I understand, no.
>
> I add the whole file just to be safe here:
>
> root@adc1:/etc/kea# cat kea-ctrl-agent.conf
> // This is a basic configuration for the Kea Control Agent.
> //
> // This is just a very basic configuration. Kea comes with large suite
> (over 30)
> // of configuration examples and extensive Kea User's Guide. Please refer to
> // those materials to get better understanding of what this software is
> able to
> // do. Comments in this configuration file sometimes refer to sections
> for more
> // details. These are section numbers in Kea User's Guide. The version
> matching
> // your software should come with your Kea package, but it is also available
> // in ISC's Knowledgebase (https://kea.readthedocs.io; the direct link for
> // the stable version is https://kea.readthedocs.io/).
> //
> // This configuration file contains only Control Agent's configuration.
> // If configurations for other Kea services are also included in this
> file they
> // are ignored by the Control Agent.
> {
>
> // This is a basic configuration for the Kea Control Agent.
> // RESTful interface to be available at http://127.0.0.1:8000/
> "Control-agent": {
>  "http-host": "10.0.0.231",
>  // If enabling HA and multi-threading, the 8000 port is used by the HA
>  // hook library http listener. When using HA hook library with
>  // multi-threading to function, make sure the port used by dedicated
>  // listener is different (e.g. 8001) than the one used by CA. Note
>  // the commands should still be sent via CA. The dedicated listener
>  // is specifically for HA updates only.
>  "http-port": 8000,
>
> "authentication": {
>  "type": "basic",
>  "realm": "kea-control-agent",
>  "clients": [
>  {
>  "user": "admin",
>  "password": "X-dEldmfRz"
>  } ]
>  },
>
>  // Specify location of the files to which the Control Agent
>  // should connect to forward commands to the DHCPv4, DHCPv6
>  // and D2 servers via unix domain sockets.
>  "control-sockets": {
>  "dhcp4": {
>  "socket-type": "unix",
>  "socket-name": "/tmp/kea4-ctrl-socket"
>  }
>  #"dhcp6":

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Darren Ankney]

Hi Stefan,

I imagine your socket file is being removed during routine OS cleaning
of /tmp/  I would not store it there in production (I do often do that
in testing, however).  If the socket file gets removed, Kea will not
recreate it until a restart.

Thank you,

Darren Ankney

On Fri, Jun 23, 2023 at 4:21 AM Stefan G. Weichinger  wrote:
>
> Am 22.06.23 um 19:35 schrieb Eric Graham:
> > Check the permissions of the control socket on your server:
> >
> > * Confirm that the Kea control agent can read it * Confirm that
> > SELinux is not denying the attempt
>
> Eric, thanks for your help.
>
> If it works most of the time ... why wouldn't the permissions be OK then?
>
> OK:
>
> root@adc1:/etc/kea# ls -l /tmp/kea4*
> srwxr-xr-x 1 _kea _kea 0 22. Jun 07:35 /tmp/kea4-ctrl-socket
> -rw--- 1 _kea _kea 0 22. Jun 07:35 /tmp/kea4-ctrl-socket.lock
>
>
> root@adc2:/etc/kea# ls -l /tmp/kea4*
> srwxr-xr-x 1 _kea _kea 0 17. Mai 08:18 /tmp/kea4-ctrl-socket
> -rw--- 1 _kea _kea 0 17. Mai 08:18 /tmp/kea4-ctrl-socket.lock
>
> SElinux: don't know how to check that right now ...
>
>
> > Your control agent on the other server isn't configured to connect to
> >  the socket for dhcp4 on your primary server, correct?
>
> As far as I understand, no.
>
> I add the whole file just to be safe here:
>
> root@adc1:/etc/kea# cat kea-ctrl-agent.conf
> // This is a basic configuration for the Kea Control Agent.
> //
> // This is just a very basic configuration. Kea comes with large suite
> (over 30)
> // of configuration examples and extensive Kea User's Guide. Please refer to
> // those materials to get better understanding of what this software is
> able to
> // do. Comments in this configuration file sometimes refer to sections
> for more
> // details. These are section numbers in Kea User's Guide. The version
> matching
> // your software should come with your Kea package, but it is also available
> // in ISC's Knowledgebase (https://kea.readthedocs.io; the direct link for
> // the stable version is https://kea.readthedocs.io/).
> //
> // This configuration file contains only Control Agent's configuration.
> // If configurations for other Kea services are also included in this
> file they
> // are ignored by the Control Agent.
> {
>
> // This is a basic configuration for the Kea Control Agent.
> // RESTful interface to be available at http://127.0.0.1:8000/
> "Control-agent": {
>  "http-host": "10.0.0.231",
>  // If enabling HA and multi-threading, the 8000 port is used by the HA
>  // hook library http listener. When using HA hook library with
>  // multi-threading to function, make sure the port used by dedicated
>  // listener is different (e.g. 8001) than the one used by CA. Note
>  // the commands should still be sent via CA. The dedicated listener
>  // is specifically for HA updates only.
>  "http-port": 8000,
>
> "authentication": {
>  "type": "basic",
>  "realm": "kea-control-agent",
>  "clients": [
>  {
>  "user": "admin",
>  "password": "X-dEldmfRz"
>  } ]
>  },
>
>  // Specify location of the files to which the Control Agent
>  // should connect to forward commands to the DHCPv4, DHCPv6
>  // and D2 servers via unix domain sockets.
>  "control-sockets": {
>  "dhcp4": {
>  "socket-type": "unix",
>  "socket-name": "/tmp/kea4-ctrl-socket"
>  }
>  #"dhcp6": {
>  #"socket-type": "unix",
>  #"socket-name": "/tmp/kea6-ctrl-socket"
>  #},
>  #"d2": {
>  #"socket-type": "unix",
>  #"socket-name": "/tmp/kea-ddns-ctrl-socket"
>  #}
>  },
>
>  // Specify hooks libraries that are attached to the Control Agent.
>  // Such hooks libraries should support 'control_command_receive'
>  // hook point. This is currently commented out because it has to
>  // point to the existing hooks library. Otherwise the Control
>  // Agent will fail to start.
>  "hooks-libraries": [
> //  {
> //  "library":
> "/usr/lib/x86_64-linux-gnu/kea/hooks/control-agent-commands.so",
> //  "parameters": {
> //  "param1": "foo"
> //  }
> //  }
>  ],
>
> // Logging configuration starts here. Kea uses different loggers to log
> various
> // activities. For details (e.g. names of loggers), see Chapter 18.
>  "loggers": [
>  {
>  // This specifies the logging for Control Agent daemon.
>  "name": "kea-ctrl-agent",
>  "output_options": [
>  {
>  // Specifies the output file. There are several special
> values
>  // supported:
>  // - stdout (prints on standard output)
>  // - stderr (prints on standard error)
>  // - syslog (logs to syslog)
>  // - syslog:name (logs to syslog using specified name)
>  

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 22.06.23 um 19:35 schrieb Eric Graham:

Check the permissions of the control socket on your server:

* Confirm that the Kea control agent can read it * Confirm that
SELinux is not denying the attempt


Eric, thanks for your help.

If it works most of the time ... why wouldn't the permissions be OK then?

OK:

root@adc1:/etc/kea# ls -l /tmp/kea4*
srwxr-xr-x 1 _kea _kea 0 22. Jun 07:35 /tmp/kea4-ctrl-socket
-rw--- 1 _kea _kea 0 22. Jun 07:35 /tmp/kea4-ctrl-socket.lock


root@adc2:/etc/kea# ls -l /tmp/kea4*
srwxr-xr-x 1 _kea _kea 0 17. Mai 08:18 /tmp/kea4-ctrl-socket
-rw--- 1 _kea _kea 0 17. Mai 08:18 /tmp/kea4-ctrl-socket.lock

SElinux: don't know how to check that right now ...



Your control agent on the other server isn't configured to connect to
 the socket for dhcp4 on your primary server, correct?


As far as I understand, no.

I add the whole file just to be safe here:

root@adc1:/etc/kea# cat kea-ctrl-agent.conf
// This is a basic configuration for the Kea Control Agent.
//
// This is just a very basic configuration. Kea comes with large suite 
(over 30)

// of configuration examples and extensive Kea User's Guide. Please refer to
// those materials to get better understanding of what this software is 
able to
// do. Comments in this configuration file sometimes refer to sections 
for more
// details. These are section numbers in Kea User's Guide. The version 
matching

// your software should come with your Kea package, but it is also available
// in ISC's Knowledgebase (https://kea.readthedocs.io; the direct link for
// the stable version is https://kea.readthedocs.io/).
//
// This configuration file contains only Control Agent's configuration.
// If configurations for other Kea services are also included in this 
file they

// are ignored by the Control Agent.
{

// This is a basic configuration for the Kea Control Agent.
// RESTful interface to be available at http://127.0.0.1:8000/
"Control-agent": {
"http-host": "10.0.0.231",
// If enabling HA and multi-threading, the 8000 port is used by the HA
// hook library http listener. When using HA hook library with
// multi-threading to function, make sure the port used by dedicated
// listener is different (e.g. 8001) than the one used by CA. Note
// the commands should still be sent via CA. The dedicated listener
// is specifically for HA updates only.
"http-port": 8000,

"authentication": {
"type": "basic",
"realm": "kea-control-agent",
"clients": [
{
"user": "admin",
"password": "X-dEldmfRz"
} ]
},

// Specify location of the files to which the Control Agent
// should connect to forward commands to the DHCPv4, DHCPv6
// and D2 servers via unix domain sockets.
"control-sockets": {
"dhcp4": {
"socket-type": "unix",
"socket-name": "/tmp/kea4-ctrl-socket"
}
#"dhcp6": {
#"socket-type": "unix",
#"socket-name": "/tmp/kea6-ctrl-socket"
#},
#"d2": {
#"socket-type": "unix",
#"socket-name": "/tmp/kea-ddns-ctrl-socket"
#}
},

// Specify hooks libraries that are attached to the Control Agent.
// Such hooks libraries should support 'control_command_receive'
// hook point. This is currently commented out because it has to
// point to the existing hooks library. Otherwise the Control
// Agent will fail to start.
"hooks-libraries": [
//  {
//  "library": 
"/usr/lib/x86_64-linux-gnu/kea/hooks/control-agent-commands.so",

//  "parameters": {
//  "param1": "foo"
//  }
//  }
],

// Logging configuration starts here. Kea uses different loggers to log 
various

// activities. For details (e.g. names of loggers), see Chapter 18.
"loggers": [
{
// This specifies the logging for Control Agent daemon.
"name": "kea-ctrl-agent",
"output_options": [
{
// Specifies the output file. There are several special 
values

// supported:
// - stdout (prints on standard output)
// - stderr (prints on standard error)
// - syslog (logs to syslog)
// - syslog:name (logs to syslog using specified name)
// Any other value is considered a name of the file
"output": "stdout",

// Shorter log pattern suitable for use with systemd,
// avoids redundant information
"pattern": "%-5p %m\n"

// This governs whether the log output is flushed to 
disk after

// every write.
// "flush": false,

// This specifies the maximum size of the file before it is
// rotated.
// "maxsize": 1048576,

// This specifies the maximum number of rotated files 

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Eric Graham]

Check the permissions of the control socket on your server:

  *   Confirm that the Kea control agent can read it
  *   Confirm that SELinux is not denying the attempt

Your control agent on the other server isn't configured to connect to the 
socket for dhcp4 on your primary server, correct?

Eric Graham
DevOps Specialist
Direct: 605.990.1859
eric.gra...@vantagepnt.com<mailto:eric.gra...@vantagepnt.com>
[cid:fa418b8e-e445-4e50-adc4-29ffc34b9590]

From: Stefan G. Weichinger 
Sent: Thursday, June 22, 2023 12:18 PM
To: Eric Graham ; kea-users@lists.isc.org 

Subject: Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork 
and dhcp4 gets lost

CAUTION: This email originated outside the organization. Do not click any links 
or attachments unless you have verified the sender.

Am 22.06.23 um 16:35 schrieb Eric Graham:
> Stefan,
>
> What version of Stork are you using? I saw a similar issue
> <https://gitlab.isc.org/isc-projects/stork/-/issues/792> in 1.7.

1.10.0

> Do your Kea control agent logs show any issues during the event?

Didn't find the time today to browse the logs.

the problematic server adc1 logs something after the event in stork.

stork displays


2023-06-20 20:01:23 daemon [2] dhcp4 is unreachable

kea-debug.log:

2023-06-20 20:00:54.374 CEST INFO
[kea-dhcp4.commands/664.140016487107008] COMMAND_RECEIVED Received
command 'subnet4-list'
2023-06-20 20:00:54.679 CEST INFO
[kea-dhcp4.commands/664.140016487107008] COMMAND_RECEIVED Received
command 'ha-heartbeat'
2023-06-20 20:02:04.789 CEST INFO
[kea-dhcp4.ha-hooks/664.140016487107008] HA_STATE_TRANSITION server
transitions from HOT-STANDBY to WAITING state, partner state is PARTNER-DOWN
2023-06-20 20:02:04.790 CEST INFO
[kea-dhcp4.ha-hooks/664.140016487107008] HA_LEASE_UPDATES_DISABLED lease
updates will not be sent to the partner while in WAITING state
2023-06-20 20:02:04.790 CEST INFO
[kea-dhcp4.ha-hooks/664.140016487107008] HA_LOCAL_DHCP_DISABLE local
DHCP service is disabled while the adc1 is in the WAITING state
2023-06-20 20:02:04.790 CEST INFO
[kea-dhcp4.ha-hooks/664.140016487107008] HA_STATE_TRANSITION server
transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN
2023-06-20 20:02:04.790 CEST INFO
[kea-dhcp4.ha-hooks/664.140016487107008] HA_LEASE_UPDATES_DISABLED lease
updates will not be sent to the partner while in SYNCING state
2023-06-20 20:02:04.790 CEST INFO
[kea-dhcp4.ha-hooks/664.140016487107008] HA_SYNC_START starting lease
database synchronization with adc2
2023-06-20 20:02:04.807 CEST INFO
[kea-dhcp4.ha-hooks/664.140016487107008]
HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 118 leases from adc2
2023-06-20 20:02:04.809 CEST INFO
[kea-dhcp4.ha-hooks/664.140016487107008] HA_SYNC_SUCCESSFUL lease
database synchronization with adc2 completed successfully in 19.611 ms
2023-06-20 20:02:04.809 CEST INFO
[kea-dhcp4.ha-hooks/664.140016487107008] HA_STATE_TRANSITION server
transitions from SYNCING to READY state, partner state is PARTNER-DOWN
2023-06-20 20:02:04.810 CEST INFO
[kea-dhcp4.ha-hooks/664.140016487107008] HA_LEASE_UPDATES_DISABLED lease
updates will not be sent to the partner while in READY state
2023-06-20 20:06:51.205 CEST INFO
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_START
starting Lease File Cleanup
2023-06-20 20:06:51.206 CEST INFO
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_EXECUTE
executing Lease File Cleanup using: /usr/sbin/kea-lfc -4 -x
/var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o
/var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed
-p /var/lib/kea/dhcp4.leases.pid -c ignored-path
2023-06-20 21:06:52.202 CEST INFO
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_START
starting Lease File Cleanup
2023-06-20 21:06:52.203 CEST INFO
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_EXECUTE
executing Lease File Cleanup using: /usr/sbin/kea-lfc -4 -x
/var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o
/var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed
-p /var/lib/kea/dhcp4.leases.pid -c ignored-path
2023-06-20 22:06:52.466 CEST INFO
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_START
starting Lease File Cleanup
2023-06-20 22:06:52.467 CEST INFO
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_EXECUTE
executing Lease File Cleanup using: /usr/sbin/kea-lfc -4 -x
/var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o
/var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed
-p /var/lib/kea/dhcp4.leases.pid -c ignored-path
2023-06-20 23:06:53.457 CEST INFO
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_START
starting Lease File Cleanup

The other server only says:

2023-06-20 20:01:04.701 CEST WARN
[kea-dhcp4.ha-hooks/291908.140125778662080] HA_HEARTBEAT_FAILED
heartbeat to adc1 (http://10.0.0.231:8000/) failed: unable to forward
command to the dhcp4 service

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Stefan G. Weichinger]

Am 22.06.23 um 16:35 schrieb Eric Graham:

Stefan,

What version of Stork are you using? I saw a similar issue 
 in 1.7.


1.10.0


Do your Kea control agent logs show any issues during the event?


Didn't find the time today to browse the logs.

the problematic server adc1 logs something after the event in stork.

stork displays


2023-06-20 20:01:23 daemon [2] dhcp4 is unreachable

kea-debug.log:

2023-06-20 20:00:54.374 CEST INFO 
[kea-dhcp4.commands/664.140016487107008] COMMAND_RECEIVED Received 
command 'subnet4-list'
2023-06-20 20:00:54.679 CEST INFO 
[kea-dhcp4.commands/664.140016487107008] COMMAND_RECEIVED Received 
command 'ha-heartbeat'
2023-06-20 20:02:04.789 CEST INFO 
[kea-dhcp4.ha-hooks/664.140016487107008] HA_STATE_TRANSITION server 
transitions from HOT-STANDBY to WAITING state, partner state is PARTNER-DOWN
2023-06-20 20:02:04.790 CEST INFO 
[kea-dhcp4.ha-hooks/664.140016487107008] HA_LEASE_UPDATES_DISABLED lease 
updates will not be sent to the partner while in WAITING state
2023-06-20 20:02:04.790 CEST INFO 
[kea-dhcp4.ha-hooks/664.140016487107008] HA_LOCAL_DHCP_DISABLE local 
DHCP service is disabled while the adc1 is in the WAITING state
2023-06-20 20:02:04.790 CEST INFO 
[kea-dhcp4.ha-hooks/664.140016487107008] HA_STATE_TRANSITION server 
transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN
2023-06-20 20:02:04.790 CEST INFO 
[kea-dhcp4.ha-hooks/664.140016487107008] HA_LEASE_UPDATES_DISABLED lease 
updates will not be sent to the partner while in SYNCING state
2023-06-20 20:02:04.790 CEST INFO 
[kea-dhcp4.ha-hooks/664.140016487107008] HA_SYNC_START starting lease 
database synchronization with adc2
2023-06-20 20:02:04.807 CEST INFO 
[kea-dhcp4.ha-hooks/664.140016487107008] 
HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 118 leases from adc2
2023-06-20 20:02:04.809 CEST INFO 
[kea-dhcp4.ha-hooks/664.140016487107008] HA_SYNC_SUCCESSFUL lease 
database synchronization with adc2 completed successfully in 19.611 ms
2023-06-20 20:02:04.809 CEST INFO 
[kea-dhcp4.ha-hooks/664.140016487107008] HA_STATE_TRANSITION server 
transitions from SYNCING to READY state, partner state is PARTNER-DOWN
2023-06-20 20:02:04.810 CEST INFO 
[kea-dhcp4.ha-hooks/664.140016487107008] HA_LEASE_UPDATES_DISABLED lease 
updates will not be sent to the partner while in READY state
2023-06-20 20:06:51.205 CEST INFO 
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_START 
starting Lease File Cleanup
2023-06-20 20:06:51.206 CEST INFO 
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_EXECUTE 
executing Lease File Cleanup using: /usr/sbin/kea-lfc -4 -x 
/var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o 
/var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed 
-p /var/lib/kea/dhcp4.leases.pid -c ignored-path
2023-06-20 21:06:52.202 CEST INFO 
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_START 
starting Lease File Cleanup
2023-06-20 21:06:52.203 CEST INFO 
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_EXECUTE 
executing Lease File Cleanup using: /usr/sbin/kea-lfc -4 -x 
/var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o 
/var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed 
-p /var/lib/kea/dhcp4.leases.pid -c ignored-path
2023-06-20 22:06:52.466 CEST INFO 
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_START 
starting Lease File Cleanup
2023-06-20 22:06:52.467 CEST INFO 
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_EXECUTE 
executing Lease File Cleanup using: /usr/sbin/kea-lfc -4 -x 
/var/lib/kea/dhcp4.leases.2 -i /var/lib/kea/dhcp4.leases.1 -o 
/var/lib/kea/dhcp4.leases.output -f /var/lib/kea/dhcp4.leases.completed 
-p /var/lib/kea/dhcp4.leases.pid -c ignored-path
2023-06-20 23:06:53.457 CEST INFO 
[kea-dhcp4.dhcpsrv/664.140016487107008] DHCPSRV_MEMFILE_LFC_START 
starting Lease File Cleanup


The other server only says:

2023-06-20 20:01:04.701 CEST WARN 
[kea-dhcp4.ha-hooks/291908.140125778662080] HA_HEARTBEAT_FAILED 
heartbeat to adc1 (http://10.0.0.231:8000/) failed: unable to forward 
command to the dhcp4 service: No such file or directory. The server is 
likely to be offline, error code 1





--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users

Re: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and dhcp4 gets lost

[Eric Graham]

Stefan,

What version of Stork are you using? I saw a similar 
issue in 1.7.

Do your Kea control agent logs show any issues during the event?

Eric Graham
DevOps Specialist
Direct: 605.990.1859
eric.gra...@vantagepnt.com
[cid:f6860e1c-74e6-4e1e-baef-95ec5de2c3bd]

From: Kea-users  on behalf of Stefan G. 
Weichinger 
Sent: Thursday, June 22, 2023 12:55 AM
To: kea-users@lists.isc.org 
Subject: [Kea-users] kea-2.2.0 - HA cluster - communication between stork and 
dhcp4 gets lost

CAUTION: This email originated outside the organization. Do not click any links 
or attachments unless you have verified the sender.

We see this in the stork events every few days:

2023-06-20 20:01:23 daemon [2] dhcp4 is unreachable

2023-06-20 20:01:07 Communication with daemon [2] dhcp4 of app
kea@10.0.0.231 failed

After a restart of both dhcp4 and stork-agent on that adc1-server things
work again.

I will have to check the logs in more detail, sure.

2 things:

1) we collect the prometheus metrics from stork and visualize them in
Grafana.

storkserver_auth_unreachable_machine_total{instance=~"$instance"}

is always 0, even when the mentioned events are seen and I would assume
that one of 2 machines should be marked unreachable. Right?

2) it's not solving the problem at the root, but I consider setting up
some external monitoring to detect this outage and let the monitoring
restart the daemons ...

I use monit (https://mmonit.com/wiki/Monit/ConfigurationExamples) for
such things, and think of letting it do http-API-calls to isc-kea to
check things.

Right approach?

thanks, Stefan
--
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users
-- 
ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.

To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users