On Mon, Nov 23, 2020 at 9:42 AM Alex K <rightkickt...@gmail.com> wrote:
>
>
> On Mon, Nov 23, 2020 at 9:35 AM Dominik Holler <dhol...@redhat.com> wrote:
>
>>
>>
>> On Fri, Nov 20, 2020 at 12:38 PM Alex K <rightkickt...@gmail.com> wrote:
>>
>>> Following the above, I was seeing that OVN provider connectivity test
>>> was failing due to some certificate issue and had to do the following to
>>> fix it:
>>>
>>> names="ovirt-provider-ovn"
>>>
>>> subject="$(\
>>> openssl x509 \
>>> -in /etc/pki/ovirt-engine/certs/apache.cer \
>>> -noout \
>>> -subject | \
>>> sed \
>>> 's;subject= \(.*\);\1;'
>>> )"
>>>
>>> . /usr/share/ovirt-engine/bin/engine-prolog.sh
>>>
>>> for name in $names; do
>>> /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
>>> --name="${name}" \
>>> --password=mypass \
>>> --subject="${subject}" \
>>> --keep-key \
>>> --san=DNS:"${ENGINE_FQDN}"
>>> done
>>>
>>> Having fixed the above, when trying to connect two VMs on some OVN
>>> logical switches it seems they are not able to reach each other.
>>> I had previously added such logical switched at engine by running:
>>>
>>> ovn-nbctl ls-add ovn-net0
>>> ovn-nbctl ls-add ovn-net1
>>> etc
>>>
>>>
>> Not related: Please use ovirt-provider-ovn to create and manage ovn
>> entities.
>>
>>
>>> Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I
>>> see:
>>> reconnect|WARN|unix#45: connection dropped (Connection reset by peer)
>>>
>>>
>> /var/log/openvswitch/ovn-controller.log might contain the reason.
>>
>>
>>> Also systemctl status ovirt-provider-ovn.service at engine shows:
>>> /usr/lib/python2.7/site-packages/urllib3/connection.py:344:
>>> SubjectAltNameWarning:...
>>>
>>>
>> Looks not good, do tou know which connection this warning referes to?
>>
>>
>>> I have restarted at engine both engine and ovn services:
>>> systemctl restart ovirt-engine
>>> systemctl status ovirt-provider-ovn.service
>>>
>>> I have also restarted the relevant service at each host:
>>> systemctl restart ovn-controller.service
>>>
>>> When running at host the following it stucks and does not give any
>>> output:
>>> ovn-sbctl show
>>>
>>>
>> This is expected, the ovn southbound and northbound db exists only on the
>> ovn-central, which is places on the same machine as oVirt Engine.
>> Only the ovn-controller, which controls openvswitch, and openvswitch,
>> which is implementing the data plane, is placed on the ovn-chassis / oVirt
>> host.
>>
>>
>>> I see that the certificate is imported at key-store as it has the same
>>> fingerprint with the previous root CA:
>>>
>>> keytool -list -alias ovirt-provider-ovn -keystore
>>> /var/lib/ovirt-engine/external_truststore
>>>
>>>
>> This is only relevant for the connection from oVirt Engine to
>> ovirt-provider-ovn.
>>
>>
>>> At this same cluster, I had previously changed the domain name of each
>>> host and engine using the rename tool.
>>> And now replaced the certificates as per previous described so as to fix
>>> the imageio cert issue and ovn issue.
>>>
>>> It seems that OVN is not happy with the status of certificates.
>>> When testing connection at engine GUI i get a prompt to trust the cert,
>>> and when pressing ok i get a green confirmation of successful connection.
>>>
>>>
>> This is only relevant for the connection from oVirt Engine to
>> ovirt-provider-ovn. The prompt to trust the certificate might be redundant.
>> If you get the green confirmation, oVirt Engine is happy and the
>> certificate of the REST API of ovirt-provider-ovn is fine.
>>
>>
>>> Is there anything else that can be done to fix OVN functionality?
>>>
>>
>> Please try to understand what is wrong in the connection between
>> ovn-controller and ovn south bound db.
>> /var/log/openvswitch/ovn-controller.log should be helpful and might
>> contain the reason.
>>
> Will run the steps again to see. Do you think I need to take additional
> steps when fixing the OVN certs issue due to domain change that this
> cluster has undergone?
>
This time was not able to make OVN provider succeed at the test connection
with the new certs. Restored the pki to its previous state, using the
previous CA. Now, I do not see any errors for the last hour. Created also
from GUI a logical switch named ovn-switch-1. Attached two VMs to it and
was not able to confirm ping between VMs.
At engine I see both the MAC addresses of each guest VM.
[root@engine ~]# ovn-nbctl show
switch ae4e03eb-e097-4629-a7bd-3272eee65599
(ovirt-ovn-switch-1-bd04ad69-11a9-46d0-b571-f7dee62dfb7c)
port c4e4098b-764f-4696-8506-ccf46a535fd2
addresses: ["00:1a:4a:16:02:59"]
port 14b8fb48-5ab1-4b10-93e1-d4e1fab17b51
addresses: ["00:1a:4a:16:02:5e"]
and the two hosts:
[root@engine ~]# ovn-sbctl show
Chassis "580a335e-f55f-4947-95d5-e90690b05125"
hostname: "v1"
Encap geneve
ip: "10.10.10.12"
options: {csum="true"}
Port_Binding "14b8fb48-5ab1-4b10-93e1-d4e1fab17b51"
Port_Binding "c4e4098b-764f-4696-8506-ccf46a535fd2"
Chassis "872e9ea7-fe6e-455f-8645-a2d6159c7552"
hostname: "v0"
Encap geneve
ip: "10.10.10.11"
options: {csum="true"}
The status of OVN provider seem fine at engine:
[root@engine ~]# systemctl status ovirt-provider-ovn.service
● ovirt-provider-ovn.service - oVirt OVN provider
Loaded: loaded (/usr/lib/systemd/system/ovirt-provider-ovn.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-11-23 16:40:33 EET; 10min ago
Main PID: 25293 (python2)
Tasks: 4
CGroup: /system.slice/ovirt-provider-ovn.service
└─25293 /usr/bin/python2
/usr/share/ovirt-provider-ovn/ovirt_provider_ovn.py
Nov 23 16:40:33 engine.mtis.tech systemd[1]: Started oVirt OVN provider.
Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - -
[23/Nov/2020 16:44:26] "POST /v2.0//tokens HTTP/1.1" 200 -
Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - -
[23/Nov/2020 16:44:26] "GET /v2.0/networks HTTP/1.1" 200 -
Also the status of OVN controller at each host seem fine:
[root@v0 ~]# systemctl status ovn-controller.service
● ovn-controller.service - OVN controller daemon
Loaded: loaded (/usr/lib/systemd/system/ovn-controller.service; enabled;
vendor preset: disabled)
Active: active (running) since Sun 2020-11-22 11:43:17 EET; 1 day 5h ago
Process: 11492 ExecStop=/usr/share/openvswitch/scripts/ovn-ctl
stop_controller (code=exited, status=0/SUCCESS)
Process: 11509 ExecStart=/usr/share/openvswitch/scripts/ovn-ctl
--no-monitor start_controller $OVN_CONTROLLER_OPTS (code=exited,
status=0/SUCCESS)
Main PID: 11539 (ovn-controller)
Tasks: 4
CGroup: /system.slice/ovn-controller.service
└─11539 ovn-controller unix:/var/run/openvswitch/db.sock
-vconsole:emer -vsyslog:err -vfile:info --private-key=/etc/pki...
Nov 22 11:43:17 v0.mtis.tech systemd[1]: Starting OVN controller daemon...
Nov 22 11:43:17 v0.mtis.tech ovn-ctl[11509]: Starting ovn-controller [ OK
]
Nov 22 11:43:17 v0.mtis.tech systemd[1]: Started OVN controller daemon.
What could be the reason of traffic not being forwarded through the logical
switch?
>>
>>
>>> Thanx
>>> Alex
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkickt...@gmail.com> wrote:
>>>
>>>> Seems that all services (imageio, ovn, web socket) are fine after
>>>> following the above and importing the new self signed CA certificate.
>>>> DId run also engine-setup as I was trying to fix the imageio cert
>>>> issue, though seems that that was only fixed after importing the CA cert at
>>>> browser and engine-setup might not be needed.
>>>>
>>>> On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkickt...@gmail.com> wrote:
>>>>
>>>>> Seems I had a typo at
>>>>> /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf.
>>>>> I will repeat the test to verify that all services are functional
>>>>> following this process.
>>>>>
>>>>> On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkickt...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I am trying to replace the ovirt certificate at ovirt 4.3 following
>>>>>> this:
>>>>>>
>>>>>>
>>>>>> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/administration_guide/appe-red_hat_enterprise_virtualization_and_ssl
>>>>>>
>>>>>> I am doing the following:
>>>>>> I have engine FQDN: manager.lab.local
>>>>>>
>>>>>> 1. Create root CA private key:
>>>>>> openssl genrsa -des3 -out root.key 2048
>>>>>>
>>>>>> 2. Generate root certificate: (enter passphrase of root key)
>>>>>> openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out
>>>>>> root.pem
>>>>>> cp root.pem /tmp
>>>>>>
>>>>>> 3. Create key and CSR for engine:
>>>>>> openssl genrsa -out manager.lab.local.key 2048
>>>>>> openssl req -new -out manager.lab.local.csr -key manager.lab.local.key
>>>>>>
>>>>>> 4. Generate a certificate for engine and sign with the root CA key:
>>>>>>
>>>>>> openssl x509 -req -in manager.lab.local.csr \
>>>>>> -CA root.pem \
>>>>>> -CAkey root.key \
>>>>>> -CAcreateserial \
>>>>>> -out manager.lab.local.crt \
>>>>>> -days 3650 \
>>>>>> -sha256 \
>>>>>> -extensions v3_req
>>>>>>
>>>>>> 5. Verify the trust chain and check the certificate details:
>>>>>> openssl verify -CAfile root.pem manager.lab.local.crt
>>>>>> openssl x509 -text -noout -in manager.lab.local.crt | head -15
>>>>>>
>>>>>> 6. Generate a P12 container: (with empty password)
>>>>>> openssl pkcs12 -export -out /tmp/apache.p12 \
>>>>>> -inkey manager.lab.local.key \
>>>>>> -in manager.lab.local.crt
>>>>>>
>>>>>> 8. Export key and cert:
>>>>>> openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key
>>>>>> openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer
>>>>>>
>>>>>> From the above steps we should have the following:
>>>>>>
>>>>>> /tmp/root.pem
>>>>>> /tmp/apache.p12
>>>>>> /tmp/apache.key
>>>>>> /tmp/apache.cer
>>>>>>
>>>>>> 9. Place the certificates:
>>>>>> hosted-engine --set-maintenance --mode=global
>>>>>> cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck
>>>>>> cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12
>>>>>> cp /tmp/root.pem /etc/pki/ca-trust/source/anchors
>>>>>> update-ca-trust
>>>>>> rm /etc/pki/ovirt-engine/apache-ca.pem
>>>>>> cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem
>>>>>>
>>>>>> Backup existing key and cert:
>>>>>> cp /etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>> /etc/pki/ovirt-engine/keys/apache.key.nopass.bck
>>>>>> cp /etc/pki/ovirt-engine/certs/apache.cer
>>>>>> /etc/pki/ovirt-engine/certs/apache.cer.bck
>>>>>> cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>> cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer
>>>>>> chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>> chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>> systemctl restart httpd.service
>>>>>>
>>>>>> 10. Create a new trust store configuration file:
>>>>>> vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf
>>>>>>
>>>>>> ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts"
>>>>>> ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
>>>>>>
>>>>>> 11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
>>>>>> :
>>>>>> vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
>>>>>>
>>>>>> SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer
>>>>>> SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>>
>>>>>> 12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf:
>>>>>> vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf
>>>>>>
>>>>>> # Key file for SSL connections
>>>>>> ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>> # Certificate file for SSL connections
>>>>>> ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer
>>>>>>
>>>>>> 13. Import the certificate at system-wide java trust store
>>>>>>
>>>>>> update-ca-trust extract
>>>>>> keytool -list -alias ovirt -keystore /etc/pki/java/cacerts
>>>>>>
>>>>>> 14. Restart services:
>>>>>> systemctl restart httpd.service
>>>>>> systemctl restart ovirt-provider-ovn.service
>>>>>> systemctl restart ovirt-imageio-proxy
>>>>>> systemctl restart ovirt-websocket-proxy
>>>>>> systemctl restart ovirt-engine.service
>>>>>>
>>>>>> Following the above I get at engine GUI:
>>>>>>
>>>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to
>>>>>> find
>>>>>> valid certification path to requested target
>>>>>>
>>>>>> I have tried also to run engine-setup in case it could fix anything
>>>>>> (it renewed the cert due to missing subjectAltName), and the above error
>>>>>> still persists.
>>>>>> I have tried several other suggestions from similar issues reported
>>>>>> at this list without any luck.
>>>>>> I have run out of ideas. Am I missing anything?
>>>>>> Thanx for any suggestions.
>>>>>> Alex
>>>>>>
>>>>> _______________________________________________
>>> Users mailing list -- users@ovirt.org
>>> To unsubscribe send an email to users-le...@ovirt.org
>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZBEOAEGVVTOLGLBFKFLGUM6/
>>>
>>
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SPC5THJTUXPZIQWKAFUUJJSG2NMWN26C/
