On Tue, Nov 24, 2020 at 8:45 AM Alex K <rightkickt...@gmail.com> wrote:
>
>
> On Mon, Nov 23, 2020 at 4:54 PM Alex K <rightkickt...@gmail.com> wrote:
>
>>
>>
>> On Mon, Nov 23, 2020 at 9:42 AM Alex K <rightkickt...@gmail.com> wrote:
>>
>>>
>>>
>>> On Mon, Nov 23, 2020 at 9:35 AM Dominik Holler <dhol...@redhat.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Fri, Nov 20, 2020 at 12:38 PM Alex K <rightkickt...@gmail.com>
>>>> wrote:
>>>>
>>>>> Following the above, I was seeing that OVN provider connectivity test
>>>>> was failing due to some certificate issue and had to do the following to
>>>>> fix it:
>>>>>
>>>>> names="ovirt-provider-ovn"
>>>>>
>>>>> subject="$(\
>>>>> openssl x509 \
>>>>> -in /etc/pki/ovirt-engine/certs/apache.cer \
>>>>> -noout \
>>>>> -subject | \
>>>>> sed \
>>>>> 's;subject= \(.*\);\1;'
>>>>> )"
>>>>>
>>>>> . /usr/share/ovirt-engine/bin/engine-prolog.sh
>>>>>
>>>>> for name in $names; do
>>>>> /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
>>>>> --name="${name}" \
>>>>> --password=mypass \
>>>>> --subject="${subject}" \
>>>>> --keep-key \
>>>>> --san=DNS:"${ENGINE_FQDN}"
>>>>> done
>>>>>
>>>>> Having fixed the above, when trying to connect two VMs on some OVN
>>>>> logical switches it seems they are not able to reach each other.
>>>>> I had previously added such logical switched at engine by running:
>>>>>
>>>>> ovn-nbctl ls-add ovn-net0
>>>>> ovn-nbctl ls-add ovn-net1
>>>>> etc
>>>>>
>>>>>
>>>> Not related: Please use ovirt-provider-ovn to create and manage ovn
>>>> entities.
>>>>
>>>>
>>>>> Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I
>>>>> see:
>>>>> reconnect|WARN|unix#45: connection dropped (Connection reset by peer)
>>>>>
>>>>>
>>>> /var/log/openvswitch/ovn-controller.log might contain the reason.
>>>>
>>>>
>>>>> Also systemctl status ovirt-provider-ovn.service at engine shows:
>>>>> /usr/lib/python2.7/site-packages/urllib3/connection.py:344:
>>>>> SubjectAltNameWarning:...
>>>>>
>>>>>
>>>> Looks not good, do tou know which connection this warning referes to?
>>>>
>>>>
>>>>> I have restarted at engine both engine and ovn services:
>>>>> systemctl restart ovirt-engine
>>>>> systemctl status ovirt-provider-ovn.service
>>>>>
>>>>> I have also restarted the relevant service at each host:
>>>>> systemctl restart ovn-controller.service
>>>>>
>>>>> When running at host the following it stucks and does not give any
>>>>> output:
>>>>> ovn-sbctl show
>>>>>
>>>>>
>>>> This is expected, the ovn southbound and northbound db exists only on
>>>> the ovn-central, which is places on the same machine as oVirt Engine.
>>>> Only the ovn-controller, which controls openvswitch, and openvswitch,
>>>> which is implementing the data plane, is placed on the ovn-chassis / oVirt
>>>> host.
>>>>
>>>>
>>>>> I see that the certificate is imported at key-store as it has the same
>>>>> fingerprint with the previous root CA:
>>>>>
>>>>> keytool -list -alias ovirt-provider-ovn -keystore
>>>>> /var/lib/ovirt-engine/external_truststore
>>>>>
>>>>>
>>>> This is only relevant for the connection from oVirt Engine to
>>>> ovirt-provider-ovn.
>>>>
>>>>
>>>>> At this same cluster, I had previously changed the domain name of each
>>>>> host and engine using the rename tool.
>>>>> And now replaced the certificates as per previous described so as to
>>>>> fix the imageio cert issue and ovn issue.
>>>>>
>>>>> It seems that OVN is not happy with the status of certificates.
>>>>> When testing connection at engine GUI i get a prompt to trust the
>>>>> cert, and when pressing ok i get a green confirmation of successful
>>>>> connection.
>>>>>
>>>>>
>>>> This is only relevant for the connection from oVirt Engine to
>>>> ovirt-provider-ovn. The prompt to trust the certificate might be redundant.
>>>> If you get the green confirmation, oVirt Engine is happy and the
>>>> certificate of the REST API of ovirt-provider-ovn is fine.
>>>>
>>>>
>>>>> Is there anything else that can be done to fix OVN functionality?
>>>>>
>>>>
>>>> Please try to understand what is wrong in the connection between
>>>> ovn-controller and ovn south bound db.
>>>> /var/log/openvswitch/ovn-controller.log should be helpful and might
>>>> contain the reason.
>>>>
>>> Will run the steps again to see. Do you think I need to take additional
>>> steps when fixing the OVN certs issue due to domain change that this
>>> cluster has undergone?
>>>
>> This time was not able to make OVN provider succeed at the test
>> connection with the new certs. Restored the pki to its previous state,
>> using the previous CA. Now, I do not see any errors for the last hour.
>> Created also from GUI a logical switch named ovn-switch-1. Attached two VMs
>> to it and was not able to confirm ping between VMs.
>>
>> At engine I see both the MAC addresses of each guest VM.
>> [root@engine ~]# ovn-nbctl show
>> switch ae4e03eb-e097-4629-a7bd-3272eee65599
>> (ovirt-ovn-switch-1-bd04ad69-11a9-46d0-b571-f7dee62dfb7c)
>> port c4e4098b-764f-4696-8506-ccf46a535fd2
>> addresses: ["00:1a:4a:16:02:59"]
>> port 14b8fb48-5ab1-4b10-93e1-d4e1fab17b51
>> addresses: ["00:1a:4a:16:02:5e"]
>>
>> and the two hosts:
>> [root@engine ~]# ovn-sbctl show
>> Chassis "580a335e-f55f-4947-95d5-e90690b05125"
>> hostname: "v1"
>> Encap geneve
>> ip: "10.10.10.12"
>> options: {csum="true"}
>> Port_Binding "14b8fb48-5ab1-4b10-93e1-d4e1fab17b51"
>> Port_Binding "c4e4098b-764f-4696-8506-ccf46a535fd2"
>> Chassis "872e9ea7-fe6e-455f-8645-a2d6159c7552"
>> hostname: "v0"
>> Encap geneve
>> ip: "10.10.10.11"
>> options: {csum="true"}
>>
>> The status of OVN provider seem fine at engine:
>> [root@engine ~]# systemctl status ovirt-provider-ovn.service
>> ● ovirt-provider-ovn.service - oVirt OVN provider
>> Loaded: loaded (/usr/lib/systemd/system/ovirt-provider-ovn.service;
>> enabled; vendor preset: disabled)
>> Active: active (running) since Mon 2020-11-23 16:40:33 EET; 10min ago
>> Main PID: 25293 (python2)
>> Tasks: 4
>> CGroup: /system.slice/ovirt-provider-ovn.service
>> └─25293 /usr/bin/python2
>> /usr/share/ovirt-provider-ovn/ovirt_provider_ovn.py
>>
>> Nov 23 16:40:33 engine.mtis.tech systemd[1]: Started oVirt OVN provider.
>> Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - -
>> [23/Nov/2020 16:44:26] "POST /v2.0//tokens HTTP/1.1" 200 -
>> Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - -
>> [23/Nov/2020 16:44:26] "GET /v2.0/networks HTTP/1.1" 200 -
>>
>> Also the status of OVN controller at each host seem fine:
>> [root@v0 ~]# systemctl status ovn-controller.service
>> ● ovn-controller.service - OVN controller daemon
>> Loaded: loaded (/usr/lib/systemd/system/ovn-controller.service;
>> enabled; vendor preset: disabled)
>> Active: active (running) since Sun 2020-11-22 11:43:17 EET; 1 day 5h
>> ago
>> Process: 11492 ExecStop=/usr/share/openvswitch/scripts/ovn-ctl
>> stop_controller (code=exited, status=0/SUCCESS)
>> Process: 11509 ExecStart=/usr/share/openvswitch/scripts/ovn-ctl
>> --no-monitor start_controller $OVN_CONTROLLER_OPTS (code=exited,
>> status=0/SUCCESS)
>> Main PID: 11539 (ovn-controller)
>> Tasks: 4
>> CGroup: /system.slice/ovn-controller.service
>> └─11539 ovn-controller unix:/var/run/openvswitch/db.sock
>> -vconsole:emer -vsyslog:err -vfile:info --private-key=/etc/pki...
>>
>> Nov 22 11:43:17 v0.mtis.tech systemd[1]: Starting OVN controller daemon...
>> Nov 22 11:43:17 v0.mtis.tech ovn-ctl[11509]: Starting ovn-controller [
>> OK ]
>> Nov 22 11:43:17 v0.mtis.tech systemd[1]: Started OVN controller daemon.
>>
>> What could be the reason of traffic not being forwarded through the
>> logical switch?
>>
> I noticed that there is a network port security setting when defining the
> network at the external provider, which is enabled by default.
>
> [image: image.png]
> Setting this to disabled made the VMs attached to the same network to be
> able to communicate through it.
>
Happy to hear.
> It seems that when enabled this does drop all traffic by default if not
> specifically configured. I am wondering where I could configure a security
> group/policy to allow such traffic.
>
>
oVirt implements a subset of the OpenStack Networking API, please report
deviant behavior as a bug, including the default behavior without explicit
configuration.
Please find the details about the network port security in
https://github.com/oVirt/ovirt-provider-ovn/blob/master/docs/provider_api_description.adoc#security
>
>>
>>
>>>>
>>>>
>>>>> Thanx
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkickt...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Seems that all services (imageio, ovn, web socket) are fine after
>>>>>> following the above and importing the new self signed CA certificate.
>>>>>> DId run also engine-setup as I was trying to fix the imageio cert
>>>>>> issue, though seems that that was only fixed after importing the CA cert
>>>>>> at
>>>>>> browser and engine-setup might not be needed.
>>>>>>
>>>>>> On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkickt...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Seems I had a typo at
>>>>>>> /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf.
>>>>>>> I will repeat the test to verify that all services are functional
>>>>>>> following this process.
>>>>>>>
>>>>>>> On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkickt...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I am trying to replace the ovirt certificate at ovirt 4.3 following
>>>>>>>> this:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/administration_guide/appe-red_hat_enterprise_virtualization_and_ssl
>>>>>>>>
>>>>>>>> I am doing the following:
>>>>>>>> I have engine FQDN: manager.lab.local
>>>>>>>>
>>>>>>>> 1. Create root CA private key:
>>>>>>>> openssl genrsa -des3 -out root.key 2048
>>>>>>>>
>>>>>>>> 2. Generate root certificate: (enter passphrase of root key)
>>>>>>>> openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out
>>>>>>>> root.pem
>>>>>>>> cp root.pem /tmp
>>>>>>>>
>>>>>>>> 3. Create key and CSR for engine:
>>>>>>>> openssl genrsa -out manager.lab.local.key 2048
>>>>>>>> openssl req -new -out manager.lab.local.csr -key
>>>>>>>> manager.lab.local.key
>>>>>>>>
>>>>>>>> 4. Generate a certificate for engine and sign with the root CA key:
>>>>>>>>
>>>>>>>> openssl x509 -req -in manager.lab.local.csr \
>>>>>>>> -CA root.pem \
>>>>>>>> -CAkey root.key \
>>>>>>>> -CAcreateserial \
>>>>>>>> -out manager.lab.local.crt \
>>>>>>>> -days 3650 \
>>>>>>>> -sha256 \
>>>>>>>> -extensions v3_req
>>>>>>>>
>>>>>>>> 5. Verify the trust chain and check the certificate details:
>>>>>>>> openssl verify -CAfile root.pem manager.lab.local.crt
>>>>>>>> openssl x509 -text -noout -in manager.lab.local.crt | head -15
>>>>>>>>
>>>>>>>> 6. Generate a P12 container: (with empty password)
>>>>>>>> openssl pkcs12 -export -out /tmp/apache.p12 \
>>>>>>>> -inkey manager.lab.local.key \
>>>>>>>> -in manager.lab.local.crt
>>>>>>>>
>>>>>>>> 8. Export key and cert:
>>>>>>>> openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key
>>>>>>>> openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer
>>>>>>>>
>>>>>>>> From the above steps we should have the following:
>>>>>>>>
>>>>>>>> /tmp/root.pem
>>>>>>>> /tmp/apache.p12
>>>>>>>> /tmp/apache.key
>>>>>>>> /tmp/apache.cer
>>>>>>>>
>>>>>>>> 9. Place the certificates:
>>>>>>>> hosted-engine --set-maintenance --mode=global
>>>>>>>> cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck
>>>>>>>> cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12
>>>>>>>> cp /tmp/root.pem /etc/pki/ca-trust/source/anchors
>>>>>>>> update-ca-trust
>>>>>>>> rm /etc/pki/ovirt-engine/apache-ca.pem
>>>>>>>> cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem
>>>>>>>>
>>>>>>>> Backup existing key and cert:
>>>>>>>> cp /etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>>>> /etc/pki/ovirt-engine/keys/apache.key.nopass.bck
>>>>>>>> cp /etc/pki/ovirt-engine/certs/apache.cer
>>>>>>>> /etc/pki/ovirt-engine/certs/apache.cer.bck
>>>>>>>> cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>>>> cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer
>>>>>>>> chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>>>> chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>>>> systemctl restart httpd.service
>>>>>>>>
>>>>>>>> 10. Create a new trust store configuration file:
>>>>>>>> vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf
>>>>>>>>
>>>>>>>> ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts"
>>>>>>>> ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
>>>>>>>>
>>>>>>>> 11. Edit
>>>>>>>> /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf :
>>>>>>>> vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
>>>>>>>>
>>>>>>>> SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer
>>>>>>>> SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>>>>
>>>>>>>> 12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf:
>>>>>>>> vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf
>>>>>>>>
>>>>>>>> # Key file for SSL connections
>>>>>>>> ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass
>>>>>>>> # Certificate file for SSL connections
>>>>>>>> ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer
>>>>>>>>
>>>>>>>> 13. Import the certificate at system-wide java trust store
>>>>>>>>
>>>>>>>> update-ca-trust extract
>>>>>>>> keytool -list -alias ovirt -keystore /etc/pki/java/cacerts
>>>>>>>>
>>>>>>>> 14. Restart services:
>>>>>>>> systemctl restart httpd.service
>>>>>>>> systemctl restart ovirt-provider-ovn.service
>>>>>>>> systemctl restart ovirt-imageio-proxy
>>>>>>>> systemctl restart ovirt-websocket-proxy
>>>>>>>> systemctl restart ovirt-engine.service
>>>>>>>>
>>>>>>>> Following the above I get at engine GUI:
>>>>>>>>
>>>>>>>> sun.security.validator.ValidatorException: PKIX path building
>>>>>>>> failed: sun.security.provider.certpath.SunCertPathBuilderException:
>>>>>>>> unable
>>>>>>>> to find valid certification path to requested target
>>>>>>>>
>>>>>>>> I have tried also to run engine-setup in case it could fix anything
>>>>>>>> (it renewed the cert due to missing subjectAltName), and the above
>>>>>>>> error
>>>>>>>> still persists.
>>>>>>>> I have tried several other suggestions from similar issues reported
>>>>>>>> at this list without any luck.
>>>>>>>> I have run out of ideas. Am I missing anything?
>>>>>>>> Thanx for any suggestions.
>>>>>>>> Alex
>>>>>>>>
>>>>>>> _______________________________________________
>>>>> Users mailing list -- users@ovirt.org
>>>>> To unsubscribe send an email to users-le...@ovirt.org
>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>>>>> oVirt Code of Conduct:
>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>> List Archives:
>>>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZBEOAEGVVTOLGLBFKFLGUM6/
>>>>>
>>>>
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/MFQLFV7YCOLRDICTWLAKL7C2DAQJHUP2/
