date:20230808

[ovirt-users] Re: Trouble restoring + upgrading to ovirt 4.5 system after host crashed

2023-08-08 Thread Yedidyah Bar David

Hi,

On Tue, Aug 8, 2023 at 9:21 PM David Johnson 
wrote:

> Good afternoon all,
>
> We had a confluence of events hit all at once and need help desperately.
> Our Ovirt engine system recently crashed and is unrecoverable. Due to a
> power maintenance event at the data center, 1/3 of our VM's are offline.
>
> I have recent backups from the engine created with engine-backup.
>

How do you run engine-backup for backups? What version? What OS?


>
> I installed a clean Centos 9 and followed the directions to install
> the ovirt-engine .
>
> After I restore the backup, the engine-setup fails on the keycloak
> configuration.
>
> *From clean system:*
>
> *Install: **(Observe failed scriptlet during install, but rom install
> still succeeds)*
>
> [root@ovirt2 administrator]# dnf install -y ovirt-engine
> Last metadata expiration check: 2:08:15 ago on Tue 08 Aug 2023 10:11:31 AM
> CDT.
> Dependencies resolved.
>
> =
>  Package  Architecture
>   VersionRepository
>  Size
>
> =
> Installing:
>  ovirt-engine noarch
>   4.5.4-1.el9centos-ovirt45
>  13 M
> Installing dependencies:
>  SuperLU  x86_64
>   5.3.0-2.el9epel
> 182 k
>
> (Snip ...)
>
>
>
>
>
> *  Running scriptlet: ovirt-vmconsole-1.0.9-1.el9.noarch
>
> 60/425Failed to resolve allow statement at
> /var/lib/selinux/targeted/tmp/modules/400/ovirt_vmconsole/cil:539Failed to
> resolve AST/usr/sbin/semodule:  Failed!*
>
>
This might cause a problem later on, but I do not think it's related to
your current issue.


>
> (Snip ...)
>  xmlrpc-common-3.1.3-1.1.el9.noarch
>xorg-x11-fonts-ISO8859-1-100dpi-7.5-33.el9.noarch
>   zziplib-0.13.71-9.el9.x86_64
>
> Complete!
>
>
> *Engine-restore (no visible issues):*
>
> [root@ovirt2 administrator]# engine-backup --mode=restore
> --log=restore1.log --file=Downloads/engine-2023-08-06.22.00.02.bak
> --provision-all-databases --restore-permissions
> Start of engine-backup with mode 'restore'
> scope: all
> archive file: Downloads/engine-2023-08-06.22.00.02.bak
> log file: restore1.log
> Preparing to restore:
> - Unpacking file 'Downloads/engine-2023-08-06.22.00.02.bak'
> Restoring:
> - Files
>
> --
> Please note:
>
> Operating system is different from the one used during backup.
> Current operating system: centos9
> Operating system at backup: centos8
>
>
I do not think this is the problem, but you might try as well on centos8.


>
> Apache httpd configuration will not be restored.
> You will be asked about it on the next engine-setup run.
>
> --
> Provisioning PostgreSQL users/databases:
> - user 'engine', database 'engine'
> - user 'ovirt_engine_history', database 'ovirt_engine_history'
> - user 'ovirt_engine_history_grafana' on database 'ovirt_engine_history'
>
>


> Restoring:
> - Engine database 'engine'
>   - Cleaning up temporary tables in engine database 'engine'
>   - Updating DbJustRestored VdcOption in engine database
>   - Resetting DwhCurrentlyRunning in dwh_history_timekeeping in engine
> database
>   - Resetting HA VM status
>
> --
> Please note:
>
> The engine database was backed up at 2023-08-06 22:00:19.0 -0500 .
>
> Objects that were added, removed or changed after this date, such as
> virtual
> machines, disks, etc., are missing in the engine, and will probably require
> recovery or recreation.
>
> --
> - DWH database 'ovirt_engine_history'
> - Grafana database '/var/lib/grafana/grafana.db'
>
>
No Keycloak DB restored. I guess it was not backed up, perhaps not even
configured.


> You should now run engine-setup.
> Done.
> [root@ovirt2 administrator]#
>
>
> *Engine-setup :*
>
> [root@ovirt2 administrator]# engine-setup
> [ INFO  ] Stage: Initializing
> [ INFO  ] Stage: Environment setup
>   Configuration files:
> /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf,
> /etc/ovirt-engine-setup.conf.d/10-packaging.conf,
>   /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf
>   Log file:
> /var/log/ovirt-engine/setup/ovirt-engine-setup-20230808124501-joveku.log
>   Version: otopi-1.10.3 (otopi-1.10.3-1.el9)
> [ INFO  ] The engine DB has been restored from a backup
>
> *[ ERROR ] Failed to execute stage 'Environment

[ovirt-users] Re: Trouble restoring + upgrading to ovirt 4.5 system after host crashed

2023-08-08 Thread David Johnson

Good evening all,

I was able to work past this by restarting the dying machine briefly,
backing up the keycloak database with pg_dump, and migrating it to the new
system.  setup-engine seems to work ok, but it is not generating the certs
for the keycloak, now.

When I attempt to log in to the web console, I got this message:

Warning alert:PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification path to requested target


engine.log records:

2023-08-08 17:56:44,507-05 INFO
>  [org.ovirt.engine.core.sso.service.NegotiateAuthService] (default task-2)
> [] User admin@ovirt@internalkeycloak-authz with profile [internalsso]
> successfully logged in with scopes : ovirt-app-admin ovirt-app-api
> ovirt-app-portal ovirt-ext=auth:sequence-priority=~
> ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search
> ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate
> ovirt-ext=token:password-access
> 2023-08-08 17:56:44,623-05 ERROR
> [org.ovirt.engine.core.aaa.servlet.SsoPostLoginServlet] (default task-2) []
> server_error: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
> 2023-08-08 17:56:50,216-05 INFO
>  [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService]
> (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1)
> [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for
> tasks.


After adding engine.cer to the java keystore and restarting the engine all
returned to normal.

Thank you!

*David Johnson*




On Tue, Aug 8, 2023 at 3:30 PM David Johnson 
wrote:

> Update:
>
> I have confirmed the original ovirt version has an ovirt_engine_keycloak
> database, but the database was not backed up by the engine-backup command
>
> *David Johnson*
> *Director of Development, Maxis Technology*
> 844.696.2947 ext 702 (o) | 479.531.3590 (c)
> 
> 
> 
>
> *Follow us:*  
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/EVFSS6RHGYC6WSRT7SL4TXQXM6PSBDA4/

[ovirt-users] Re: Certificates expired...

2023-08-08 Thread cen


Hi,

I went through a similar ordeal half a year ago and forgot all the exact 
procedures already but for me, in the end after following all the guides 
and replacing the "standard" certs


it was either engine.p12 or apache.p12 keystore that also had outdated 
certs (apparently mTLS is being used!).


Updating these keystores is not documented anywhere. No idea if you are 
in the same situation but wanted to throw this out there.


Best regards, cen

On 4. 08. 23 20:12, Jason P. Thomas wrote:
I updated the VDSM certs on the hosts and the apache cert on the 
engine.  I'm guessing something is wrong with however the engine 
interacts with vdsm, I just don't know exactly what to do about it.


Jason

On 8/4/23 14:00, Derek Atkins wrote:

Sounds like the Host Certs need to be updated.. Or possibly even the
Engine CA Cert.

-derek

On Fri, August 4, 2023 1:45 pm, Jason P. Thomas wrote:

Konstantin,
Right after I sent the email I got the engine running.  The
libvirt-spice certs had incorrect ownership.  It still is not 
connecting

to anything.  Error in Events on the Engine is now: "VDSM
 command Get Host Capabilities failed: General SSLEngine
problem"

So status right now is, all VMs are running.  Engine web ui is
accessible.  Engine shows all hosts as unassigned or Connecting or
NonResponsive with repeated entries of the above error in Events.

Sincerely,
Jason

On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote:

Now the engine won't start at all and I'm afraid I'm one power outage
away from complete disaster.  I need to keep the old location up and
functioning for another 4-6 months, so any insights would be greatly
appreciated.

Hi,

'engine won't start at all' can mean two things:

1) OS can't boot and thus you can't do SSH. Assuming that we are 
talking
self-hosted engine, then you need to use command like below on host 
that

runs ovengine VM (virsh -c
qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list
and hosted-engine --vm-status might be helpful, VM should at least 
start

to boot in order for you to achieve connectivity via console):
hosted-engine --add-console-password --password=somepassword
and then connect via VNC to IP that you will see in output and 
password

that you used

2) ovirt-engine service can't start
In that case it is likely that you will find reason of that in
   journalctl -u ovirt-engine --no-pager
(/var/log/ovirt-engine/engine.log)

BR,
Konstantin
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/ 


___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/ 






___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3GFW2SRSZB5QHNY3ABXG2KPQ6ZA36M5I/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AMVZEWY45QHPEDHJQZGJMZWESN2RZBPB/

[ovirt-users] Trouble restoring + upgrading to ovirt 4.5 system after host crashed

2023-08-08 Thread David Johnson

Update:

I have confirmed the original ovirt version has an ovirt_engine_keycloak
database, but the database was not backed up by the engine-backup command

*David Johnson*
*Director of Development, Maxis Technology*
844.696.2947 ext 702 (o) | 479.531.3590 (c)




*Follow us:*  
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/7CIBF64MKNV263ORIQL3XFTRGW62CCI4/

[ovirt-users] Trouble restoring + upgrading to ovirt 4.5 system after host crashed

2023-08-08 Thread David Johnson

Good afternoon all,

We had a confluence of events hit all at once and need help desperately.
Our Ovirt engine system recently crashed and is unrecoverable. Due to a
power maintenance event at the data center, 1/3 of our VM's are offline.

I have recent backups from the engine created with engine-backup.

I installed a clean Centos 9 and followed the directions to install
the ovirt-engine .

After I restore the backup, the engine-setup fails on the keycloak
configuration.

*From clean system:*

*Install: **(Observe failed scriptlet during install, but rom install still
succeeds)*

[root@ovirt2 administrator]# dnf install -y ovirt-engine
Last metadata expiration check: 2:08:15 ago on Tue 08 Aug 2023 10:11:31 AM
CDT.
Dependencies resolved.
=
 Package  Architecture
  VersionRepository
 Size
=
Installing:
 ovirt-engine noarch
  4.5.4-1.el9centos-ovirt45
 13 M
Installing dependencies:
 SuperLU  x86_64
  5.3.0-2.el9epel
182 k

(Snip ...)





*  Running scriptlet: ovirt-vmconsole-1.0.9-1.el9.noarch

60/425Failed to resolve allow statement at
/var/lib/selinux/targeted/tmp/modules/400/ovirt_vmconsole/cil:539Failed to
resolve AST/usr/sbin/semodule:  Failed!*

(Snip ...)
 xmlrpc-common-3.1.3-1.1.el9.noarch
 xorg-x11-fonts-ISO8859-1-100dpi-7.5-33.el9.noarch
  zziplib-0.13.71-9.el9.x86_64

Complete!


*Engine-restore (no visible issues):*

[root@ovirt2 administrator]# engine-backup --mode=restore
--log=restore1.log --file=Downloads/engine-2023-08-06.22.00.02.bak
--provision-all-databases --restore-permissions
Start of engine-backup with mode 'restore'
scope: all
archive file: Downloads/engine-2023-08-06.22.00.02.bak
log file: restore1.log
Preparing to restore:
- Unpacking file 'Downloads/engine-2023-08-06.22.00.02.bak'
Restoring:
- Files
--
Please note:

Operating system is different from the one used during backup.
Current operating system: centos9
Operating system at backup: centos8

Apache httpd configuration will not be restored.
You will be asked about it on the next engine-setup run.
--
Provisioning PostgreSQL users/databases:
- user 'engine', database 'engine'
- user 'ovirt_engine_history', database 'ovirt_engine_history'
- user 'ovirt_engine_history_grafana' on database 'ovirt_engine_history'
Restoring:
- Engine database 'engine'
  - Cleaning up temporary tables in engine database 'engine'
  - Updating DbJustRestored VdcOption in engine database
  - Resetting DwhCurrentlyRunning in dwh_history_timekeeping in engine
database
  - Resetting HA VM status
--
Please note:

The engine database was backed up at 2023-08-06 22:00:19.0 -0500 .

Objects that were added, removed or changed after this date, such as virtual
machines, disks, etc., are missing in the engine, and will probably require
recovery or recreation.
--
- DWH database 'ovirt_engine_history'
- Grafana database '/var/lib/grafana/grafana.db'
You should now run engine-setup.
Done.
[root@ovirt2 administrator]#


*Engine-setup :*

[root@ovirt2 administrator]# engine-setup
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
  Configuration files:
/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf,
/etc/ovirt-engine-setup.conf.d/10-packaging.conf,
  /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf
  Log file:
/var/log/ovirt-engine/setup/ovirt-engine-setup-20230808124501-joveku.log
  Version: otopi-1.10.3 (otopi-1.10.3-1.el9)
[ INFO  ] The engine DB has been restored from a backup

*[ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to
Keycloak database 'ovirt_engine_keycloak' using existing credentials:
ovirt_engine_keycloak@localhost:5432*[ INFO  ] Stage: Clean up
  Log file is located at
/var/log/ovirt-engine/setup/ovirt-engine-setup-20230808124501-joveku.log
[ INFO  ] Generating answer file
'/var/lib/ovirt-engine/setup/answers/20230808124504-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination


*[ ERROR ] Execution of setup failed[root@ovirt2 administrator]#*


*Engine-cleanup results:*
(snip)

[ INFO  ] Stage: Clean up
  Log file is located at

[ovirt-users] Re: Trouble restoring + upgrading to ovirt 4.5 system after host crashed

[ovirt-users] Re: Trouble restoring + upgrading to ovirt 4.5 system after host crashed

[ovirt-users] Re: Certificates expired...

[ovirt-users] Trouble restoring + upgrading to ovirt 4.5 system after host crashed

[ovirt-users] Trouble restoring + upgrading to ovirt 4.5 system after host crashed

5 matches

Site Navigation

Mail list logo

Footer information