[ovirt-users] stomp timeout in org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient

[Chris Smith]

Hi all,

I'm randomly getting this in the engine logs;

These times are UTC+1 so are different to the ovirt host logs.

2024-05-06 21:10:47,238+01 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connection timeout for host '10.10.3.36', last response arrived 1604 ms
ago.
2024-05-06 21:10:51,750+01 INFO
 [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to /10.10.3.36
2024-05-06 21:11:18,994+01 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connection timeout for host '10.10.3.36', last response arrived 1885 ms
ago.
2024-05-06 21:11:22,197+01 INFO
 [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connecting to /10.10.3.36
2024-05-06 21:11:27,767+01 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)
[] Connection timeout for host '10.10.3.36', last response arrived 1779 ms
ago.

and I'm trying to debug why it can't connect.

The vdsm logs on the ovirt host don't give me many clues about what's
happening. I see the stomp requests;

These times are UTC;

2024-05-06 20:10:51,807+ INFO  (Reactor thread)
[ProtocolDetector.Detector] Detected protocol stomp from :::
10.10.3.125:36296 (protocoldetector:125)
2024-05-06 20:10:51,807+ INFO  (Reactor thread) [Broker.StompAdapter]
Processing CONNECT request (stompserver:95)
2024-05-06 20:10:51,808+ INFO  (JsonRpc (StompReactor))
[Broker.StompAdapter] Subscribe command received (stompserver:124)
2024-05-06 20:10:51,808+ INFO  (JsonRpc (StompReactor))
[Broker.StompAdapter] Subscribe command received (stompserver:124)
2024-05-06 20:11:23,240+ INFO  (Reactor thread)
[ProtocolDetector.Detector] Detected protocol stomp from :::
10.10.3.125:36302 (protocoldetector:125)
2024-05-06 20:11:23,416+ INFO  (JsonRpc (StompReactor))
[Broker.StompAdapter] Processing CONNECT request (stompserver:95)
2024-05-06 20:11:23,505+ INFO  (JsonRpc (StompReactor))
[Broker.StompAdapter] Subscribe command received (stompserver:124)
2024-05-06 20:11:23,505+ INFO  (JsonRpc (StompReactor))
[Broker.StompAdapter] Subscribe command received (stompserver:124)

but there's nothing around these entries to tell me what's causing the
timeout.

Is there a script or command I can run to simulate what's happening here to
try and figure out what's going on?

Thanks,
-- 
Chris Smith
Hosting Systems Administrator
P   +61 2 9045 2800
E  csm...@squiz.net

Level 1, 435a Kent Street, Sydney NSW 2000
www.squiz.net

-- 
Squiz named leader GigaOm's Digital Experience Platform Radar report, 
2023-24. Read about the report 
<https://www.squiz.net/blog/squiz-dxp-platform-leader-in-gigaom-radar-report>!
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/2FF5Q46L7J7BOXRG7K5DB5BV6JAHR3WX/

[ovirt-users] Re: Repo for mirrorlist.ovirt.org not working

[Chris Smith]

 > >
> > > > > > Is this some scheduled downtime or is it an error?
> > > > >
> > > > >
> > > > > ___
> > > > > Users mailing list -- users@ovirt.org
> > > > > To unsubscribe send an email to users-le...@ovirt.org
> > > > > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > > > > oVirt Code of Conduct:
> > > > > https://www.ovirt.org/community/about/community-guidelines/
> > > > > List Archives:
> > > > >
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/PPT5E5224LGNAOVGCHR4KPDKJVPRNNG3/
> > > >
> > > ___
> > > Users mailing list -- users@ovirt.org
> > > To unsubscribe send an email to users-le...@ovirt.org
> > > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > > oVirt Code of Conduct:
> > > https://www.ovirt.org/community/about/community-guidelines/
> > > List Archives:
> > >
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZARL6GG5X34VVODRPIXGI6PTHWDMQVZK/
> >
> >
> > --
> > Sandro Bonazzola
> > MANAGER, SOFTWARE ENGINEERING
> > Red Hat In-Vehicle Operating System
> > Red Hat EMEA
> > Red Hat respects your work life balance. Therefore there is no need
> > to answer this email out of your office hours.
> >
> >
> > ___
> > Users mailing list -- users@ovirt.org
> > To unsubscribe send an email to users-le...@ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct:
> > https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> >
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/Q5TORRZX647MQSM6INGCREGLLBFWTSZD/
>
> --
> Nathanaël Blanchet
>
> Administrateur Systèmes et Réseaux
> Service Informatique et REseau (SIRE)
> Département des systèmes d'information
> 227 avenue Professeur-Jean-Louis-Viala
> 34193 MONTPELLIER CEDEX 5
> Tél. 33 (0)4 67 54 84 55
> Fax  33 (0)4 67 54 84 14
> blanc...@abes.fr
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/2W42ZFQAZVFQOCP2XCUPBJFG37GAZYNP/
>


-- 
Chris Smith
Hosting Systems Administrator
P   +61 2 9045 2800
E  csm...@squiz.net

Level 1, 435a Kent Street, Sydney NSW 2000
www.squiz.net

-- 
Squiz named market leader (second year) in Omdia's Selecting a Digital 
Experience Management Solution report, 2022-23. Download the report today! 
<https://www.squiz.net/omdia-report-2022-23?utm_source=Website&utm_medium=Download&utm_campaign=Omdia>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4VOYIIO6VQM2MZZ4V6S6GBTREBMZADAO/

[ovirt-users] ssl stomp reactor timeout

[Chris Smith]

Hi all,

I'm using ovirt 4.2 so I'm a little behind, but I'm trying to understand
what this error message means.

ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
Reactor) [] Connection timeout for host 'x.x.x.x', last response arrived
1549 ms ago.

Is this initiated by the engine server or the ovirt host ?
Is there a way for me to replicate what ovirt is doing manually via a
telnet or curl type command ?

Thanks,
-- 
Chris

-- 
Squiz named market leader (second year) in Omdia's Selecting a Digital 
Experience Management Solution report, 2022-23. Download the report today! 

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/Y4RKKKUKXCABEFBMWWFW6FVTT7XOVFTV/

[ovirt-users] struggling with 4.5.1 install

[Chris Smith]

Hello,

Here's the general gist of what I've done so far:


graphical install from el8 and el9 iso, using as few modifications as
possible with dhcp networking.

3.6TB of disk space for ovirt install dest.

system comes up, dashboard is online, i try to deploy hosted engine, no
dice.

[image: image.png]

it's similar to this:
https://bugzilla.redhat.com/show_bug.cgi?id=1946095

ok, so i go for cli install

i manually install ovirt-engine-appliance and that seems to go ok

i kick off ovirt-hosted-engine-setup and follow the prompts

i watch the logs and such and seems to be ok

i have to make entry in /etc/hosts to have new hostname for not yet built
hosted engine VM to resolve to IP address ( I don't have "real" DNS at the
moment)

that succeeds but then new engine VM is spun up in new IP subnet
192.168.222.0/24 that is different from primary 192.168.1.0/24 not sure why.

i am at a point where i can't get a GUI to the hosted engine IP because
it's only available to the internal adapter on the node.

i feel like the installers in the el8 and el9 iso's is just completely
b0rked.

what am I doing wrong?

oh, and where is everyone at in the IRC chat room?

thanks,

Chris
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/R43ATEJ3CDMAIUPSIT5QBTAZ54DLPOUK/

Re: [Users] so, what do you want next in oVirt?

[Chris Smith]

What about using ATA over ethernet (AoE) as an optional storage transport
within ovirt?  Is this feasible?  It seems that the protocol is fast and
efficient.
On Sep 10, 2013 11:58 AM, "Baptiste AGASSE" <
baptiste.aga...@lyra-network.com> wrote:

> Hi all,
>
> - Mail original -
> > De: "Itamar Heim" 
> > À: users@ovirt.org
> > Envoyé: Mardi 20 Août 2013 23:19:16
> > Objet: [Users] so, what do you want next in oVirt?
> >
> > earlier in the year we did a survey for feature requests /
> > improvements
> > / etc.
> >
> > since a lot of things were added, and priorities usually change, I'd
> > like to ask again for "what do you need the most from oVirt / what
> > are
> > your pain points" next?
> >
> > below[1] I've listed my understanding of what already went in from
> > previous survey requests (to various degrees of coverage).
> >
> > Thanks,
> > Itamar
> >
> > [1] from the top 12
> > V Allow disk resize
> > V Integrate Nagios/Zabbix monitoring - via a ui plugin
> > V Highly Available engine - via hosted engine[2]
> > V Open vSwitch integration - via neutron integration
> > X Allow cloning VMs without template
> > ? Enable hypervisor upgrade/updates through engine[3]
> > V Allow engine on an oVirt hosted VM - via hosted engine[2]
> > V Enable guest configuration (root password, SSH keys, network) via
> >guest agent in engine - via cloud-init
> > X Integrate v2v into engine
> > ? Bond/extend ovirtmgmt with a second network for HA/increased
> >bandwidth[4]
> > X Integrate scheduling of snapshots and VM export for backups in
> >engine[5]
> > V Spice – support Google Chrome - via mime based launch
> >
> >
> > Other items mentioned in previous survey which should be covered by
> > now:
> > - Fix timeout when adding local host during all-in-one configuration
> > - Fix engine set-up when SELinux is disabled
> > - Provide packages for el6 (CentOS, Red Hat Enterprise Linux)
> > - Allow multiple VMs to be deployed from the same template at the
> > same
> >time
> > - ISO domains on local/GlusterS
> > - Show IP addresses in Virtual Machines->Network Interfaces
> > - OpenStack Quantum support (now called Neutron)
> > - noVNC support
> > - Support spice.html5 and websocket proxy
> > - Add other guest OSes to list
> > - Port oVirt guest agent to Ubuntu[6]
> > - SLA - Allow resource time-sharing
> > - Spice - Mac client (via mime based launch)
> > - Spice - port XPI plug-in to Windows (not sure this will happen, but
> >mime based launch allows using firefox now)
> > - Spice - client for Ubuntu/Debian (should be covered via mime based
> >launch)
> >
> >
> > [2] hosted engine is in active development, but not released yet.
> > [3] host update is supported, but not for general yum update.
> > [4] a lot of improvements were done in this space, but i'm not sure
> > if
> >  they cover this exact use case
> > [5] backup api is now being pushed to master, and orchestration of
> >  backups should probably happen via 3rd part backup vendors?
> > [6] I'm not sure packaging exists yet, but ubuntu is covered for the
> >  basic functionality of the guest agent.
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
> Thanks for this thread !
>
> - ISCSI EqualLogic SAN support or use standard iscsi tools/configuration
> - SSO for webui and cli (IPA integration)
> - PXE boot for nodes
> - VMs dependencies on startup
>
> Have a nice day.
>
> Regards.
>
> ---
> Baptiste
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] so, what do you want next in oVirt?

[Chris Smith]

What about using ATA over ethernet (AoE) as an optional storage transport
within ovirt?  Is this feasible?  It seems that the protocol is fast
efficient and cleaner than than fcoe.

Thanks for considering it as a possible feature request.
On Sep 10, 2013 11:58 AM, "Baptiste AGASSE" <
baptiste.aga...@lyra-network.com> wrote:

> Hi all,
>
> - Mail original -
> > De: "Itamar Heim" 
> > À: users@ovirt.org
> > Envoyé: Mardi 20 Août 2013 23:19:16
> > Objet: [Users] so, what do you want next in oVirt?
> >
> > earlier in the year we did a survey for feature requests /
> > improvements
> > / etc.
> >
> > since a lot of things were added, and priorities usually change, I'd
> > like to ask again for "what do you need the most from oVirt / what
> > are
> > your pain points" next?
> >
> > below[1] I've listed my understanding of what already went in from
> > previous survey requests (to various degrees of coverage).
> >
> > Thanks,
> > Itamar
> >
> > [1] from the top 12
> > V Allow disk resize
> > V Integrate Nagios/Zabbix monitoring - via a ui plugin
> > V Highly Available engine - via hosted engine[2]
> > V Open vSwitch integration - via neutron integration
> > X Allow cloning VMs without template
> > ? Enable hypervisor upgrade/updates through engine[3]
> > V Allow engine on an oVirt hosted VM - via hosted engine[2]
> > V Enable guest configuration (root password, SSH keys, network) via
> >guest agent in engine - via cloud-init
> > X Integrate v2v into engine
> > ? Bond/extend ovirtmgmt with a second network for HA/increased
> >bandwidth[4]
> > X Integrate scheduling of snapshots and VM export for backups in
> >engine[5]
> > V Spice – support Google Chrome - via mime based launch
> >
> >
> > Other items mentioned in previous survey which should be covered by
> > now:
> > - Fix timeout when adding local host during all-in-one configuration
> > - Fix engine set-up when SELinux is disabled
> > - Provide packages for el6 (CentOS, Red Hat Enterprise Linux)
> > - Allow multiple VMs to be deployed from the same template at the
> > same
> >time
> > - ISO domains on local/GlusterS
> > - Show IP addresses in Virtual Machines->Network Interfaces
> > - OpenStack Quantum support (now called Neutron)
> > - noVNC support
> > - Support spice.html5 and websocket proxy
> > - Add other guest OSes to list
> > - Port oVirt guest agent to Ubuntu[6]
> > - SLA - Allow resource time-sharing
> > - Spice - Mac client (via mime based launch)
> > - Spice - port XPI plug-in to Windows (not sure this will happen, but
> >mime based launch allows using firefox now)
> > - Spice - client for Ubuntu/Debian (should be covered via mime based
> >launch)
> >
> >
> > [2] hosted engine is in active development, but not released yet.
> > [3] host update is supported, but not for general yum update.
> > [4] a lot of improvements were done in this space, but i'm not sure
> > if
> >  they cover this exact use case
> > [5] backup api is now being pushed to master, and orchestration of
> >  backups should probably happen via 3rd part backup vendors?
> > [6] I'm not sure packaging exists yet, but ubuntu is covered for the
> >  basic functionality of the guest agent.
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
> Thanks for this thread !
>
> - ISCSI EqualLogic SAN support or use standard iscsi tools/configuration
> - SSO for webui and cli (IPA integration)
> - PXE boot for nodes
> - VMs dependencies on startup
>
> Have a nice day.
>
> Regards.
>
> ---
> Baptiste
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] which file system for shared disk?

[Chris Smith]

Why not use gluster with xfs on the storage bricks?

http://www.gluster.org/

On Wed, Jul 10, 2013 at 7:15 AM, Piotr Szubiakowski
 wrote:
> Hi,
> we are developing an application where would be great if multiple host could
> have access to the same disk. I think that we can use features like shared
> disk or direct LUN to attach the same storage to multiple VM's. However to
> provide concurrent access to the resource, there should be a cluster file
> system used. The most popular open source cluster file systems are GFS2 and
> OCFS2. So my questions are:
>
> 1) Does anyone have share disk between VM's in oVirt? What fs did You used?
> 2) Is it possible to use GFS2 on VM's that are running on oVirt? Does anyone
> have run fencing mechanism with ovirt/libvirt?
>
> Many thanks,
> Piotr
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] How to rescue storage domain structure

[Chris Smith]

   119.99 GiB
  Current LE 30718
  Segments   2
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device   253:2

  --- Logical volume ---
  LV Path/dev/vg_voyager/lv_var
  LV Namelv_var
  VG Namevg_voyager
  LV UUIDserQHO-uSog-ci5m-Xx7B-AElf-GTqi-HYCRY6
  LV Write Accessread/write
  LV Creation host, time voyager, 2012-11-11 01:55:01 -0500
  LV Status  available
  # open 1
  LV Size15.00 GiB
  Current LE 3840
  Segments   2
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device   253:3

  --- Logical volume ---
  LV Path/dev/vg_voyager/lv_root
  LV Namelv_root
  VG Namevg_voyager
  LV UUIDKc43IB-5EWZ-N05E-FrN5-NgcQ-kWTv-LxfSig
  LV Write Accessread/write
  LV Creation host, time voyager, 2012-11-11 01:55:05 -0500
  LV Status  available
  # open 1
  LV Size4.00 GiB
  Current LE 1024
  Segments   1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device   253:0

  --- Logical volume ---
  LV Path/dev/vg_voyager/lv_home
  LV Namelv_home
  VG Namevg_voyager
  LV UUIDF7aJrw-FqwN-yML2-7bbX-kcuQ-12pX-1QG8Gp
  LV Write Accessread/write
  LV Creation host, time voyager, 2012-11-11 01:55:09 -0500
  LV Status  available
  # open 1
  LV Size8.00 GiB
  Current LE 2048
  Segments   1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device   253:4

  --- Logical volume ---
  LV Path/dev/vg_voyager/lv_swap
  LV Namelv_swap
  VG Namevg_voyager
  LV UUIDS5uYT4-Q3x4-3icm-SEFW-yZVW-DhLl-vSkcLc
  LV Write Accessread/write
  LV Creation host, time voyager, 2012-11-11 01:55:14 -0500
  LV Status  available
  # open 1
  LV Size3.94 GiB
  Current LE 1008
  Segments   1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device   253:1

  --- Logical volume ---
  LV Path/dev/vg_voyager/lv_tmp
  LV Namelv_tmp
  VG Namevg_voyager
  LV UUID2QeEXe-7zpq-0yLV-NT0u-9ZgY-mk8w-30n2Nz
  LV Write Accessread/write
  LV Creation host, time voyager, 2012-11-11 01:55:14 -0500
  LV Status  available
  # open 1
  LV Size8.00 GiB
  Current LE 2048
  Segments   1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device   253:5

On Mon, Apr 22, 2013 at 3:53 PM, Joop  wrote:
> Chris Smith wrote:
>>
>> List,
>>
>> I have lost the ability to manage the hosts or VM's using ovirt engine
>> web interface.  The data center is offline, and I
>> can't actually perform any operations with the hosts or VM's.  I don't
>> think that there
>> are any actions I can perform in the web interface at all.
>>
>> What's odd is that I can tell the host to go into maintenance mode
>> using the ovirt-engine web interface and it seems to go into
>> maintenance mode.  It even shows the wrench icon next to the host.  I
>> can also try and activate it after it susposedly goes into maintenance
>> mode, and It states that the host was activated, but the host never
>> actually comes up or contends for SPM status, and the data center
>> never comes online.
>>
>> >From the logs it seems that at least PKI is broken between the engine
>> and the hosts as I see numerous certificate errors on both the
>> ovirt-engine and clients.
>>
>> vdsm.log shows:
>>
>> Traceback (most recent call last):
>>   File "/usr/lib64/python2.7/SocketServer.py", line 582, in
>> process_request_thread
>> self.finish_request(request, client_address)
>>   File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
>> line 66, in finish_request
>> request.do_handshake()
>>   File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
>> self._sslobj.do_handshake()
>> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
>> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>>
>> and engine.log shows:
>>
>> 2013-04-18 18:42:43,632 ERROR
>> [org.ovirt.e

[Users] How to rescue storage domain structure

[Chris Smith]

List,

I have lost the ability to manage the hosts or VM's using ovirt engine
web interface.  The data center is offline, and I
can't actually perform any operations with the hosts or VM's.  I don't
think that there
are any actions I can perform in the web interface at all.

What's odd is that I can tell the host to go into maintenance mode
using the ovirt-engine web interface and it seems to go into
maintenance mode.  It even shows the wrench icon next to the host.  I
can also try and activate it after it susposedly goes into maintenance
mode, and It states that the host was activated, but the host never
actually comes up or contends for SPM status, and the data center
never comes online.

>From the logs it seems that at least PKI is broken between the engine
and the hosts as I see numerous certificate errors on both the
ovirt-engine and clients.

vdsm.log shows:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/SocketServer.py", line 582, in
process_request_thread
self.finish_request(request, client_address)
  File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
line 66, in finish_request
request.do_handshake()
  File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
routines:SSL3_READ_BYTES:sslv3 alert certificate unknown

and engine.log shows:

2013-04-18 18:42:43,632 ERROR
[org.ovirt.engine.core.
engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-68) Failed to decryptData must start with zero
2013-04-18 18:42:43,642 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
(QuartzScheduler_Worker-68) XML RPC error in command


Alon Bar-Lev was able to offer several good pointers in another thread
titled "Certificates and PKI seem to be broken after yum update" and
eventually concluded that the installation seems to be corrupted more
than just the certificates, truststore, and keystore, and suggested
that I start a new thread to ask about how to rescue the storage
domain structure.

The storage used for the data center is ISCSI, which is intact and
working.  In fact 2 of the VM's are still online and running on one of
the original FC17 hosts systems.

I'm not able to reinstall any of the existing hosts from the ovirt-engine web
interface.  I attempted to reinstall one of the hosts (not the SPM)
which failed.

I also tried to bring up a new, third host and add it to the cluster.
I setup another Fedora 17 box up and tried to add it to the
cluster, but it states that there are no available servers in the
cluster to probe the new host.

This is a test environment that I would like to fix, but I'm also
willing to just run engine cleanup and start over.

That said, there are 3 VM's that I would like to keep.  Two are online
and running, and I'm able to see them with virsh on that host.  I was
wondering about using virsh to backup these vm's.

The third VM exists in the database, and was set to run on the host
that I attempted to reinstall, but that VM isn't running, and when I
use virsh on it's host, virsh can't seem to find it, when I perform
the list commands, and I can't start it with virsh 

What is the best way to proceed?  It seems like it would be easier to
export the VM's using virsh from the host that they run on if
possible, then update ovirt to the latest version, recreate everything
and then import the VM's back in to the new environment.

Will this work?  Is there a procedure I can follow to do this?

Here's some additional information about the installed ovirt packages
on the ovirt-engine

[root@reliant yum.repos.d]# yum list installed | grep ovirt
ovirt-engine.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-backend.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-cli.noarch3.2.0.5-1.fc17   @updates
ovirt-engine-config.noarch 3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-dbscripts.noarch  3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-genericapi.noarch 3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-notification-
service.noarch
   3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-restapi.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-sdk.noarch3.2.0.2-1.fc17   @updates
ovirt-engine-setup.noarch  3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-tools-common.noarch   3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-userportal.noarch 3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-webadmin-portal.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-image-uploader.noarch3.1.0-0.git9c42c8.fc17
 @ovirt-stable
ovirt-iso-uploader.noarch  3.1.0-0.git1841d9.fc17
 @ovirt-stable
ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17
 @ovirt-stable
ovirt-release-fedora.noarch4-2
 @/ovirt-release-fedora.noarch
___
Users mailing list
Users@ovirt.org

Re: [Users] Certificates and PKI seem to be broken after yum update

[Chris Smith]

Since I'm not able to reinstall the host from the ovirt-engine web
interface, as another thought I wanted to see if I could bring up a
third host and add it to the cluster.
I have a host Fedora 17 box ready to go but I can't add it to the
cluster.  It states that there are no available server in the cluster
to probe the new host.

What about approaching it from the other direction.  Would I be able
to stand up an ovirt-h node on the same hardware and then add it to
ovirt from the host itself, using the setup menu?

Could it then obtain spm status and bring the storage domain online?

On Thu, Apr 18, 2013 at 7:20 PM, Chris Smith  wrote:
> engine.log attached
>
> On Thu, Apr 18, 2013 at 7:11 PM, Alon Bar-Lev  wrote:
>> Need to know precise error, please attach engine.log.
>>
>>
>> ----- Original Message -
>>> From: "Chris Smith" 
>>> To: "Alon Bar-Lev" 
>>> Cc: Users@ovirt.org
>>> Sent: Friday, April 19, 2013 2:03:59 AM
>>> Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
>>>
>>> So as of now, I can put the host into maintenance mode using the
>>> ovirt-engine web interface.  I can also try and activate it.  It
>>> states that the host was activated.   The host never actually comes up
>>> or contends for SPM status, and the data center never actually comes
>>> online.
>>>
>>> If I put the host into maintenance mode and try to reinstall it, it
>>> throws an error and size must be between 0 and 50.
>>>
>>> On Thu, Apr 18, 2013 at 6:51 PM, Alon Bar-Lev  wrote:
>>> > I am not sure I understand the status.
>>> >
>>> > Everything is working or not.
>>> > If not, what exactly fails?
>>> > Why do you run it 'again'?
>>> >
>>> > What happens if you reinstall host? Go to maintenance and select 
>>> > reinstall?
>>> >
>>> > I cannot understand how all this results from upgrade, something had
>>> > changed, the CA certificate installed on the host is probably not the CA
>>> > certificate of the engine.
>>> >
>>> > - Original Message -
>>> >> From: "Chris Smith" 
>>> >> To: "Alon Bar-Lev" , Users@ovirt.org
>>> >> Sent: Friday, April 19, 2013 1:45:23 AM
>>> >> Subject: Re: [Users] Certificates and PKI seem to be broken after yum
>>> >> update
>>> >>
>>> >> On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith 
>>> >> wrote:
>>> >> > I made a backup of the .truststore, and then followed the steps and
>>> >> > then rebooted both the ovirt-engine and one of the hosts, and
>>> >> > everything worked properly.
>>> >> >
>>> >> > If I run it again, or enter the wrong password it throws an error
>>> >> > about the key store already existing, or that the password was wrong
>>> >> > so I'm pretty sure it's good.
>>> >> >
>>> >> > vdsm.log on the host still shows:
>>> >> >
>>> >> > Traceback (most recent call last):
>>> >> >   File "/usr/lib64/python2.7/SocketServer.py", line 582, in
>>> >> > process_request_thread
>>> >> > self.finish_request(request, client_address)
>>> >> >   File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
>>> >> > line 66, in finish_request
>>> >> > request.do_handshake()
>>> >> >   File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
>>> >> > self._sslobj.do_handshake()
>>> >> > SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
>>> >> > routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>>> >> >
>>> >> > engine.log on the host shows:
>>> >> >
>>> >> > 2013-04-18 18:42:43,632 ERROR
>>> >> > [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>> >> > (QuartzScheduler_Worker-68) Failed to decryptData must start with zero
>>> >> > 2013-04-18 18:42:43,642 ERROR
>>> >> > [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>> >> > (QuartzScheduler_Worker-68) XML RPC error in command
>>> >> > GetCapabilitiesVDS ( Vds: transporter ), the error was:
>>> >> > java.util.concurrent.ExecutionException:
>&g

Re: [Users] Certificates and PKI seem to be broken after yum update

[Chris Smith]

So as of now, I can put the host into maintenance mode using the
ovirt-engine web interface.  I can also try and activate it.  It
states that the host was activated.   The host never actually comes up
or contends for SPM status, and the data center never actually comes
online.

If I put the host into maintenance mode and try to reinstall it, it
throws an error and size must be between 0 and 50.

On Thu, Apr 18, 2013 at 6:51 PM, Alon Bar-Lev  wrote:
> I am not sure I understand the status.
>
> Everything is working or not.
> If not, what exactly fails?
> Why do you run it 'again'?
>
> What happens if you reinstall host? Go to maintenance and select reinstall?
>
> I cannot understand how all this results from upgrade, something had changed, 
> the CA certificate installed on the host is probably not the CA certificate 
> of the engine.
>
> - Original Message -
>> From: "Chris Smith" 
>> To: "Alon Bar-Lev" , Users@ovirt.org
>> Sent: Friday, April 19, 2013 1:45:23 AM
>> Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
>>
>> On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith  wrote:
>> > I made a backup of the .truststore, and then followed the steps and
>> > then rebooted both the ovirt-engine and one of the hosts, and
>> > everything worked properly.
>> >
>> > If I run it again, or enter the wrong password it throws an error
>> > about the key store already existing, or that the password was wrong
>> > so I'm pretty sure it's good.
>> >
>> > vdsm.log on the host still shows:
>> >
>> > Traceback (most recent call last):
>> >   File "/usr/lib64/python2.7/SocketServer.py", line 582, in
>> > process_request_thread
>> > self.finish_request(request, client_address)
>> >   File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
>> > line 66, in finish_request
>> > request.do_handshake()
>> >   File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
>> > self._sslobj.do_handshake()
>> > SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
>> > routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>> >
>> > engine.log on the host shows:
>> >
>> > 2013-04-18 18:42:43,632 ERROR
>> > [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>> > (QuartzScheduler_Worker-68) Failed to decryptData must start with zero
>> > 2013-04-18 18:42:43,642 ERROR
>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>> > (QuartzScheduler_Worker-68) XML RPC error in command
>> > GetCapabilitiesVDS ( Vds: transporter ), the error was:
>> > java.util.concurrent.ExecutionException:
>> > java.lang.reflect.InvocationTargetException,
>> > SunCertPathBuilderException: unable to find valid certification path
>> > to requested target
>> >
>> >
>> > On Thu, Apr 18, 2013 at 4:06 AM, Alon Bar-Lev  wrote:
>> >>
>> >> You should ask these question in separate thread so people may pick them
>> >> up.
>> >>
>> >> For the .truststore, try to remove it and then execute:
>> >>
>> >> # rm -f /etc/pki/ovirt-engine/.truststore
>> >> # keytool -import -noprompt -trustcacerts -alias cacert -keypass mypass
>> >> -file /etc/pki/ovirt-engine/certs/ca.der -keystore
>> >> /etc/pki/ovirt-engine/.truststore -storepass mypass
>> >> # chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore
>> >>
>> >> It should recreate the truststore with the ca certificate you have.
>> >>
>> >> - Original Message -
>> >>> From: "Chris Smith" 
>> >>> To: "Alon Bar-Lev" 
>> >>> Cc: Users@ovirt.org
>> >>> Sent: Thursday, April 18, 2013 7:18:27 AM
>> >>> Subject: Re: [Users] Certificates and PKI seem to be broken after yum
>> >>> update
>> >>>
>> >>> If it would be easier than re-setting up the certificates, I'm also
>> >>> willing to just start over and rebuild, but I would like to export the
>> >>> VM's I have first.
>> >>> One of them is a spacewalk server, another runs DNS, and DHCP for my
>> >>> test network, and I have an asterisk server.  I would like to avoid
>> >>> having to re-create all of them.
>> >>>
>> >>> The VM's are up and running now, so I could export all of the
>> >>&

Re: [Users] Certificates and PKI seem to be broken after yum update

[Chris Smith]

On Thu, Apr 18, 2013 at 6:44 PM, Chris Smith  wrote:
> I made a backup of the .truststore, and then followed the steps and
> then rebooted both the ovirt-engine and one of the hosts, and
> everything worked properly.
>
> If I run it again, or enter the wrong password it throws an error
> about the key store already existing, or that the password was wrong
> so I'm pretty sure it's good.
>
> vdsm.log on the host still shows:
>
> Traceback (most recent call last):
>   File "/usr/lib64/python2.7/SocketServer.py", line 582, in
> process_request_thread
> self.finish_request(request, client_address)
>   File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
> line 66, in finish_request
> request.do_handshake()
>   File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
> self._sslobj.do_handshake()
> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>
> engine.log on the host shows:
>
> 2013-04-18 18:42:43,632 ERROR
> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
> (QuartzScheduler_Worker-68) Failed to decryptData must start with zero
> 2013-04-18 18:42:43,642 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
> (QuartzScheduler_Worker-68) XML RPC error in command
> GetCapabilitiesVDS ( Vds: transporter ), the error was:
> java.util.concurrent.ExecutionException:
> java.lang.reflect.InvocationTargetException,
> SunCertPathBuilderException: unable to find valid certification path
> to requested target
>
>
> On Thu, Apr 18, 2013 at 4:06 AM, Alon Bar-Lev  wrote:
>>
>> You should ask these question in separate thread so people may pick them up.
>>
>> For the .truststore, try to remove it and then execute:
>>
>> # rm -f /etc/pki/ovirt-engine/.truststore
>> # keytool -import -noprompt -trustcacerts -alias cacert -keypass mypass 
>> -file /etc/pki/ovirt-engine/certs/ca.der -keystore 
>> /etc/pki/ovirt-engine/.truststore -storepass mypass
>> # chown ovirt:ovirt /etc/pki/ovirt-engine/.truststore
>>
>> It should recreate the truststore with the ca certificate you have.
>>
>> - Original Message -
>>> From: "Chris Smith" 
>>> To: "Alon Bar-Lev" 
>>> Cc: Users@ovirt.org
>>> Sent: Thursday, April 18, 2013 7:18:27 AM
>>> Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
>>>
>>> If it would be easier than re-setting up the certificates, I'm also
>>> willing to just start over and rebuild, but I would like to export the
>>> VM's I have first.
>>> One of them is a spacewalk server, another runs DNS, and DHCP for my
>>> test network, and I have an asterisk server.  I would like to avoid
>>> having to re-create all of them.
>>>
>>> The VM's are up and running now, so I could export all of the
>>> configurations / backup the file systems, etc.
>>>
>>> Preferably I could export the VM's to an NFS export domain, or a
>>> mounted NFS share so that I can import them to the new storage domain,
>>> after I run engine-cleanup and get everything set back up.  Is there
>>> an easy way to do this?  Is it possible to create and attach an NFS
>>> export domain directly from the CLI without access to the ovirt
>>> manager without communication between the manager and hosts due to the
>>> pki issue?  Can I export the VM's directly from the hosts to a
>>> standard NFS share?
>>>
>>> Is there an equivalent xml and image file for the VM?
>>>
>>> My storage domain is iscsi and is served out from another server over
>>> 4 bonded 1 Gbps copper links.
>>>
>>>
>>>
>>> On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith  wrote:
>>> > I checked the .truststore on the ovirt engine, and it seems fine.
>>> >
>>> > [root@reliant ovirt-engine]# ls -l .truststore
>>> > -rwxr-x---. 1 ovirt ovirt 918 Apr  6 21:56 .truststore
>>> >
>>> > It's not zero bytes anyway.
>>> >
>>> > It's also the same size as the .truststore in the ovirt engine backups.
>>> >
>>> > [root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l
>>> > {} \;
>>> > -rwxr-x---. 1 ovirt ovirt 918 Aug 26  2012
>>> > ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
>>> > -rwxr-x---. 1 root root 918 Mar 24 12:42
>>> > ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-20

Re: [Users] Certificates and PKI seem to be broken after yum update

[Chris Smith]

If it would be easier than re-setting up the certificates, I'm also
willing to just start over and rebuild, but I would like to export the
VM's I have first.
One of them is a spacewalk server, another runs DNS, and DHCP for my
test network, and I have an asterisk server.  I would like to avoid
having to re-create all of them.

The VM's are up and running now, so I could export all of the
configurations / backup the file systems, etc.

Preferably I could export the VM's to an NFS export domain, or a
mounted NFS share so that I can import them to the new storage domain,
after I run engine-cleanup and get everything set back up.  Is there
an easy way to do this?  Is it possible to create and attach an NFS
export domain directly from the CLI without access to the ovirt
manager without communication between the manager and hosts due to the
pki issue?  Can I export the VM's directly from the hosts to a
standard NFS share?

Is there an equivalent xml and image file for the VM?

My storage domain is iscsi and is served out from another server over
4 bonded 1 Gbps copper links.



On Wed, Apr 17, 2013 at 11:46 PM, Chris Smith  wrote:
> I checked the .truststore on the ovirt engine, and it seems fine.
>
> [root@reliant ovirt-engine]# ls -l .truststore
> -rwxr-x---. 1 ovirt ovirt 918 Apr  6 21:56 .truststore
>
> It's not zero bytes anyway.
>
> It's also the same size as the .truststore in the ovirt engine backups.
>
> [root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l {} 
> \;
> -rwxr-x---. 1 ovirt ovirt 918 Aug 26  2012
> ./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
> -rwxr-x---. 1 root root 918 Mar 24 12:42
> ./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
>
> I haven't looked at the installCA.sh script yet.
>
> On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev  wrote:
>> This error means that the /etc/pki/ovirt-engine/.truststore is unreadable or 
>> does not contain the /etc/pki/ovirt-engine/ca.pem certificate.
>>
>> Unfortunately, the pki administration is weak in current implementation, you 
>> can trace the installation script and checkout the calls to installCA.sh to 
>> how to reproduce, please note that password are encrypted in database using 
>> the private key locate in .keystore so if you are to re-generate anything 
>> remember to keep the engine private key.
>>
>> However, if you succeed in login, the remaining problem you have is the 
>> .truststore permissions and/or content.
>>
>> Regards,
>> Alon Bar-Lev.
>>
>> - Original Message -
>>> From: "Chris Smith" 
>>> To: "Alon Bar-Lev" 
>>> Cc: Users@ovirt.org
>>> Sent: Monday, April 8, 2013 9:46:46 AM
>>> Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
>>>
>>> After setting the .keystore owner and group owner to ovirt, and
>>> rebooting, I now have a new error in engine.log
>>>
>>> 2013-04-08 02:39:16,787 ERROR
>>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>>> (QuartzScheduler_Worker-95) Failed to decryptData must start with zero
>>> 2013-04-08 02:39:16,845 ERROR
>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>>> (QuartzScheduler_Worker-95) XML RPC error in command
>>> GetCapabilitiesVDS ( Vds: transporter ), the error was:
>>> java.util.concurrent.ExecutionException:
>>> java.lang.reflect.InvocationTargetException,
>>> SunCertPathBuilderException: unable to find valid certification path
>>> to requested target
>>>
>>> Are there other files that may have been affected that I can also
>>> correct ownership or permissions on?
>>>
>>> On the host side, I get certificate unknown in vdsm.log
>>>
>>>   File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
>>> self._sslobj.do_handshake()
>>> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
>>> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>>> Thread-757809::ERROR::2013-04-08
>>> 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client
>>> ('172.16.23.8', 54489)
>>> Traceback (most recent call last):
>>>   File "/usr/lib64/python2.7/SocketServer.py", line 582, in
>>> process_request_thread
>>> self.finish_request(request, client_address)
>>>   File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
>>> line 66, in finish_request
>>> request.do_handshake()
>>>   File "/usr/lib64/python2.7/ssl.py

Re: [Users] Certificates and PKI seem to be broken after yum update

[Chris Smith]

I checked the .truststore on the ovirt engine, and it seems fine.

[root@reliant ovirt-engine]# ls -l .truststore
-rwxr-x---. 1 ovirt ovirt 918 Apr  6 21:56 .truststore

It's not zero bytes anyway.

It's also the same size as the .truststore in the ovirt engine backups.

[root@reliant ovirt-engine-backups]# find ./ -name .truststore -exec ls -l {} \;
-rwxr-x---. 1 ovirt ovirt 918 Aug 26  2012
./ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore
-rwxr-x---. 1 root root 918 Mar 24 12:42
./ovirt-engine-2013_03_24_11_15_19/ovirt-engine-2013_03_23_03_09_09/ovirt-engine/.truststore

I haven't looked at the installCA.sh script yet.

On Mon, Apr 8, 2013 at 2:58 AM, Alon Bar-Lev  wrote:
> This error means that the /etc/pki/ovirt-engine/.truststore is unreadable or 
> does not contain the /etc/pki/ovirt-engine/ca.pem certificate.
>
> Unfortunately, the pki administration is weak in current implementation, you 
> can trace the installation script and checkout the calls to installCA.sh to 
> how to reproduce, please note that password are encrypted in database using 
> the private key locate in .keystore so if you are to re-generate anything 
> remember to keep the engine private key.
>
> However, if you succeed in login, the remaining problem you have is the 
> .truststore permissions and/or content.
>
> Regards,
> Alon Bar-Lev.
>
> - Original Message -
>> From: "Chris Smith" 
>> To: "Alon Bar-Lev" 
>> Cc: Users@ovirt.org
>> Sent: Monday, April 8, 2013 9:46:46 AM
>> Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
>>
>> After setting the .keystore owner and group owner to ovirt, and
>> rebooting, I now have a new error in engine.log
>>
>> 2013-04-08 02:39:16,787 ERROR
>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>> (QuartzScheduler_Worker-95) Failed to decryptData must start with zero
>> 2013-04-08 02:39:16,845 ERROR
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>> (QuartzScheduler_Worker-95) XML RPC error in command
>> GetCapabilitiesVDS ( Vds: transporter ), the error was:
>> java.util.concurrent.ExecutionException:
>> java.lang.reflect.InvocationTargetException,
>> SunCertPathBuilderException: unable to find valid certification path
>> to requested target
>>
>> Are there other files that may have been affected that I can also
>> correct ownership or permissions on?
>>
>> On the host side, I get certificate unknown in vdsm.log
>>
>>   File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
>> self._sslobj.do_handshake()
>> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
>> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>> Thread-757809::ERROR::2013-04-08
>> 02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client
>> ('172.16.23.8', 54489)
>> Traceback (most recent call last):
>>   File "/usr/lib64/python2.7/SocketServer.py", line 582, in
>> process_request_thread
>> self.finish_request(request, client_address)
>>   File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
>> line 66, in finish_request
>> request.do_handshake()
>>   File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
>> self._sslobj.do_handshake()
>> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
>> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>>
>> Is there a procedure for just re-establishing PKI and certs for the
>> engine and hosts?
>>
>> On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev  wrote:
>> >
>> > OK... you are running a very old version of engine (3.1).
>> >
>> > The upgrade did not upgraded into 3.2, so nothing as far as I know should
>> > have been changed.
>> >
>> > But the .keystore permissions is owned by root now, so some other package
>> > (maybe selinux-policy) changed permissions...
>> >
>> > The simplest way to test is to:
>> > # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1
>> > # chown -R ovirt:ovirt /etc/pki/ovirt-engine
>> >
>> > But if that file permissions was changed, I can only assume other files
>> > were also changes...
>> >
>> > Regards,
>> > Alon
>> >
>> > - Original Message -
>> >> From: "Chris Smith" 
>> >> To: "Alon Bar-Lev" 
>> >> Cc: Users@ovirt.org
>> >> Sent: Sunday, April 7, 2013 11:51:17 AM
>> >> Subject: Re: [Users] Certificates and PKI seem to be broken afte

Re: [Users] Certificates and PKI seem to be broken after yum update

[Chris Smith]

After setting the .keystore owner and group owner to ovirt, and
rebooting, I now have a new error in engine.log

2013-04-08 02:39:16,787 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-95) Failed to decryptData must start with zero
2013-04-08 02:39:16,845 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
(QuartzScheduler_Worker-95) XML RPC error in command
GetCapabilitiesVDS ( Vds: transporter ), the error was:
java.util.concurrent.ExecutionException:
java.lang.reflect.InvocationTargetException,
SunCertPathBuilderException: unable to find valid certification path
to requested target

Are there other files that may have been affected that I can also
correct ownership or permissions on?

On the host side, I get certificate unknown in vdsm.log

  File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
Thread-757809::ERROR::2013-04-08
02:44:05,424::SecureXMLRPCServer::73::root::(handle_error) client
('172.16.23.8', 54489)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/SocketServer.py", line 582, in
process_request_thread
self.finish_request(request, client_address)
  File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
line 66, in finish_request
request.do_handshake()
  File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
routines:SSL3_READ_BYTES:sslv3 alert certificate unknown

Is there a procedure for just re-establishing PKI and certs for the
engine and hosts?

On Sun, Apr 7, 2013 at 4:58 AM, Alon Bar-Lev  wrote:
>
> OK... you are running a very old version of engine (3.1).
>
> The upgrade did not upgraded into 3.2, so nothing as far as I know should 
> have been changed.
>
> But the .keystore permissions is owned by root now, so some other package 
> (maybe selinux-policy) changed permissions...
>
> The simplest way to test is to:
> # cp -a /etc/pki/ovirt-engine /etc/pki/ovirt-engine.backup1
> # chown -R ovirt:ovirt /etc/pki/ovirt-engine
>
> But if that file permissions was changed, I can only assume other files were 
> also changes...
>
> Regards,
> Alon
>
> - Original Message -
>> From: "Chris Smith" 
>> To: "Alon Bar-Lev" 
>> Cc: Users@ovirt.org
>> Sent: Sunday, April 7, 2013 11:51:17 AM
>> Subject: Re: [Users] Certificates and PKI seem to be broken after yum update
>>
>> I did a yum update and rebooted.
>>
>> engine-upgrade was run on 24-March
>>
>> When run now, it states that there are no updates available.
>>
>> [root@reliant ~]# engine-upgrade
>> Loaded plugins: versionlock
>> Checking for updates... (This may take several minutes)
>> No updates available
>>
>>
>> [root@reliant ovirt-engine]# cat ovirt-engine-upgrade_2013_03_24_12_04_06.log
>> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
>> pgpass file, fetching DB host value
>> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
>> pgpass file, fetching DB port value
>> 2013-03-24 12:04:06::DEBUG::common_utils::585::root:: found existing
>> pgpass file, fetching DB admin value
>> 2013-03-24 12:04:07::DEBUG::engine-upgrade::302::root:: Yum list updates
>> started
>> 2013-03-24 12:04:07::DEBUG::engine-upgrade::273::root:: Yum unlock started
>> 2013-03-24 12:04:07::DEBUG::engine-upgrade::285::root:: Yum unlock
>> completed successfully
>> 2013-03-24 12:04:07::DEBUG::engine-upgrade::308::root:: Getting list
>> of packages to upgrade
>> 2013-03-24 12:04:27::DEBUG::engine-upgrade::260::root:: Yum lock started
>> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>> command --> '/bin/rpm -q ovirt-engine'
>> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>> ovirt-engine-3.1.0-4.fc17.noarch
>>
>> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>> command --> '/bin/rpm -q ovirt-engine-backend'
>> 2013-03-24 12:04:27::DEBUG::common_utils::335::root:: output =
>> ovirt-engine-backend-3.1.0-4.fc17.noarch
>>
>> 2013-03-24 12:04:27::DEBUG::common_utils::336::root:: stderr =
>> 2013-03-24 12:04:27::DEBUG::common_utils::337::root:: retcode = 0
>> 2013-03-24 12:04:27::DEBUG::common_utils::309::root:: Executing
>> command --> '/bin/rpm -q ovirt-engine-config&#x

Re: [Users] Certificates and PKI seem to be broken after yum update

[Chris Smith]

grade::320::root:: No packages
marked for update
2013-03-24 12:04:28::DEBUG::engine-upgrade::324::root:: Installed packages:
2013-03-24 12:04:28::DEBUG::engine-upgrade::325::root::
['ovirt-engine-3.1.0-4.fc17.noarch',
'ovirt-engine-backend-3.1.0-4.fc17.noarch',
'ovirt-engine-config-3.1.0-4.fc17.noarch',
'ovirt-engine-dbscripts-3.1.0-4.fc17.noarch',
'ovirt-engine-genericapi-3.1.0-4.fc17.noarch',
'ovirt-engine-notification-service-3.1.0-4.fc17.noarch',
'ovirt-engine-restapi-3.1.0-4.fc17.noarch',
'ovirt-engine-setup-3.1.0-4.fc17.noarch',
'ovirt-engine-tools-common-3.1.0-4.fc17.noarch',
'ovirt-engine-userportal-3.1.0-4.fc17.noarch',
'ovirt-engine-webadmin-portal-3.1.0-4.fc17.noarch',
'ovirt-image-uploader-3.1.0-0.git9c42c8.fc17.noarch',
'ovirt-iso-uploader-3.1.0-0.git1841d9.fc17.noarch',
'ovirt-log-collector-3.1.0-0.git10d719.fc17.noarch',
'vdsm-bootstrap-4.10.0-13.fc17.noarch']
2013-03-24 12:04:28::DEBUG::engine-upgrade::327::root:: Yum list
updated completed successfully
2013-03-24 12:04:28::DEBUG::engine-upgrade::609::root:: No updates available


Here's what's installed.

[root@reliant yum.repos.d]# yum list installed | grep ovirt
ovirt-engine.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-backend.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-cli.noarch3.2.0.5-1.fc17   @updates
ovirt-engine-config.noarch 3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-dbscripts.noarch  3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-genericapi.noarch 3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-notification-service.noarch
   3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-restapi.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-sdk.noarch3.2.0.2-1.fc17   @updates
ovirt-engine-setup.noarch  3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-tools-common.noarch   3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-userportal.noarch 3.1.0-4.fc17
 @ovirt-stable
ovirt-engine-webadmin-portal.noarch3.1.0-4.fc17
 @ovirt-stable
ovirt-image-uploader.noarch3.1.0-0.git9c42c8.fc17
 @ovirt-stable
ovirt-iso-uploader.noarch  3.1.0-0.git1841d9.fc17
 @ovirt-stable
ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17
 @ovirt-stable
ovirt-release-fedora.noarch4-2
 @/ovirt-release-fedora.noarch

On Sun, Apr 7, 2013 at 2:16 AM, Alon Bar-Lev  wrote:
> How exactly did you upgrade?
>
> Usually yum upgrade will not touch ovirt-engine packages as it is in yum 
> version lock.
> From which version to which version have you upgraded?
> Have you run engine-upgrade utility?
> If you did not, please run it.
> If you did, please attach logs from 
> /var/log/ovirt-engine/ovirt-engine-upgrade*
>
> Thanks!
>
> - Original Message -
>> From: "Chris Smith" 
>> To: Users@ovirt.org
>> Sent: Sunday, April 7, 2013 5:09:46 AM
>> Subject: [Users] Certificates and PKI seem to be broken after yum update
>>
>> I have lost the ability to manage the hosts or VM's using ovirt
>> engine web interface after performing yum update on the ovirt-engine
>> host, and on one Fedora 17 host.  The data center is offline, and I
>> can't place the hosts into maintenance mode.  I don't think that there
>> are any actions I can perform in the web interface at all.
>>
>> From the logs it seems that PKI is broken between the engine and the hosts.
>>
>> I am wondering how I can restore or re-generate all of the
>> certificates and get the hosts communicating with the ovirt-engine
>> again so that I can bring the data center back online.
>>
>> I found this page which deals with changing the engine hostname, and
>> thus re-creating the certificates and keystore on the ovirt-engine
>> node, and was wondering if this could help.  Could I follow this
>> process but keep the same hostname for the ovirt-engine node?
>>
>> http://wiki.ovirt.org/How_to_change_engine_host_name
>>
>> Currently I have 3 VM's running on two hosts.  The VM's are up, but I
>> can't do anything with them in ovirt-engine.
>>
>>
>> Here's the latest activity from engine.log from the ovirt-engine node:
>>
>> 2013-04-06 21:58:47,472 ERROR
>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>> (QuartzScheduler_Worker-61) Failed to
>> decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
>> (Permission denied)
>> 2013-04-06 21:58:47,478 ERROR
>> [org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
>> (QuartzScheduler_Worker-

[Users] Certificates and PKI seem to be broken after yum update

[Chris Smith]

I have lost the ability to manage the hosts or VM's using ovirt
engine web interface after performing yum update on the ovirt-engine
host, and on one Fedora 17 host.  The data center is offline, and I
can't place the hosts into maintenance mode.  I don't think that there
are any actions I can perform in the web interface at all.

>From the logs it seems that PKI is broken between the engine and the hosts.

I am wondering how I can restore or re-generate all of the
certificates and get the hosts communicating with the ovirt-engine
again so that I can bring the data center back online.

I found this page which deals with changing the engine hostname, and
thus re-creating the certificates and keystore on the ovirt-engine
node, and was wondering if this could help.  Could I follow this
process but keep the same hostname for the ovirt-engine node?

http://wiki.ovirt.org/How_to_change_engine_host_name

Currently I have 3 VM's running on two hosts.  The VM's are up, but I
can't do anything with them in ovirt-engine.


Here's the latest activity from engine.log from the ovirt-engine node:

2013-04-06 21:58:47,472 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-61) Failed to
decryptjava.io.FileNotFoundException: /etc/pki/ovirt-engine/.keystore
(Permission denied)
2013-04-06 21:58:47,478 ERROR
[org.ovirt.engine.core.engineencryptutils.EncryptionUtils]
(QuartzScheduler_Worker-62) Can't load keystore from file
"/etc/pki/ovirt-engine/.keystore".: java.io.FileNotFoundException:
/etc/pki/ovirt-engine/.keystore (Permission denied)
at java.io.FileInputStream.open(Native Method) [rt.jar:1.7.0_09-icedtea]
at java.io.FileInputStream.(FileInputStream.java:138)
[rt.jar:1.7.0_09-icedtea]
at 
org.ovirt.engine.core.engineencryptutils.EncryptionUtils.getKeyStore(EncryptionUtils.java:214)
[engine-encryptutils.jar:]
at 
org.ovirt.engine.core.engineencryptutils.EncryptionUtils.decrypt(EncryptionUtils.java:139)
[engine-encryptutils.jar:]
at 
org.ovirt.engine.core.dao.VdsStaticDAODbFacadeImpl.decryptPassword(VdsStaticDAODbFacadeImpl.java:139)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:253)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl$VdsRowMapper.mapRow(VdsDAODbFacadeImpl.java:169)
[engine-dal.jar:]
at 
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:653)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:702)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121)
[engine-dal.jar:]
at 
org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:164)
[spring-jdbc-2.5.6.SEC02.jar:2.5.6.SEC02]
at 
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:36)
[engine-dal.jar:]
at 
org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:31)
[engine-dal.jar:]
at 
org.ovirt.engine.core.vdsbroker.VdsManager$1.runInTransaction(VdsManager.java:219)
[engine-vdsbroker.jar:]
at 
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168)
[engine-utils.jar:]
at 
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107)
[engine-utils.jar:]
at 
org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:215)
[engine-vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown
Source)