On Fri, Dec 24, 2021 at 1:06 PM Marcin Sobczyk <msobc...@redhat.com> wrote: > > Hi All, > > OST currently fails all the time during engine setup. > Here's a piece of ansible log that's seen repeatedly and I think > describes the problem: > > 11:07:54 E "engine-config", > 11:07:54 E "-s", > 11:07:54 E "OvfUpdateIntervalInMinutes=10" > 11:07:54 E ], > 11:07:54 E "delta": "0:00:01.142926", > 11:07:54 E "end": "2021-12-24 11:06:37.894810", > 11:07:54 E "invocation": { > 11:07:54 E "module_args": { > 11:07:54 E "_raw_params": "engine-config -s > OvfUpdateIntervalInMinutes='10' ", > 11:07:54 E "_uses_shell": false, > 11:07:54 E "argv": null, > 11:07:54 E "chdir": null, > 11:07:54 E "creates": null, > 11:07:54 E "executable": null, > 11:07:54 E "removes": null, > 11:07:54 E "stdin": null, > 11:07:54 E "stdin_add_newline": true, > 11:07:54 E "strip_empty_ends": true, > 11:07:54 E "warn": false > 11:07:54 E } > 11:07:54 E }, > 11:07:54 E "item": { > 11:07:54 E "key": "OvfUpdateIntervalInMinutes", > 11:07:54 E "value": "10" > 11:07:54 E }, > 11:07:54 E "msg": "non-zero return code", > 11:07:54 E "rc": 1, > 11:07:54 E "start": "2021-12-24 11:06:36.751884", > 11:07:54 E "stderr": "Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false", > 11:07:54 E "stderr_lines": [ > 11:07:54 E "Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false" > 11:07:54 E ], > 11:07:54 E "stdout": "Error loading module from > /usr/share/ovirt-engine/modules/common/org/springframework/main/module.xml", > 11:07:54 E "stdout_lines": [ > 11:07:54 E "Error loading module from > /usr/share/ovirt-engine/modules/common/org/springframework/main/module.xml" > > We do set some config values for the engine in OST when running > engine-setup. I tried commenting these out, but then engine failed > health check anyway: > > "Status code was 503 and not [200]: HTTP Error 503: Service Unavailable" > > The last working set of OST images was the one from Dec 23, 2021 2:05:08 > AM. The first broken one is from Dec 24, 2021 2:05:09 AM. The shipped > ovirt-engine RPMs doesn't seem to contain any important changes for > these two sets, but AFAICS the newer ovirt-dependencies RPM did take in > a couple of patches that look suspicious [1][2][3]. The patches were > merged on November 16th, but it seems they were first used in that > broken set from Dec 24 (the one from Dec 23 seems to contain > ovirt-dependencies RPM based on this [4] commit). > > I wanted to try out an older version of ovirt-dependencies, but I think > they were wiped out from resources.ovirt.org. > > I will disable cyclic el8stream OST runs for now, cause all of them > fail. If there's anyone working and able to make a build with those > patches reverted and test that out, please ping me and I'll re-enable them. > > Regards, Marcin > > [1] https://gerrit.ovirt.org/c/ovirt-dependencies/+/114699 > [2] https://gerrit.ovirt.org/c/ovirt-dependencies/+/113877
This is the one that broke us - it replaced a set of shell scripts with maven. These scripts also created: # ls -l /usr/share/java/ovirt-dependencies/4.4 total 0 lrwxrwxrwx. 1 root root 24 Nov 4 18:00 gwt-servlet.jar -> ../gwt-servlet-2.9.0.jar lrwxrwxrwx. 1 root root 31 Nov 4 18:00 spring-aop.jar -> ../spring-aop-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 33 Nov 4 18:00 spring-beans.jar -> ../spring-beans-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 35 Nov 4 18:00 spring-context.jar -> ../spring-context-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 32 Nov 4 18:00 spring-core.jar -> ../spring-core-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 38 Nov 4 18:00 spring-expression.jar -> ../spring-expression-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 38 Nov 4 18:00 spring-instrument.jar -> ../spring-instrument-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 32 Nov 4 18:00 spring-jdbc.jar -> ../spring-jdbc-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 32 Nov 4 18:00 spring-test.jar -> ../spring-test-5.0.4.RELEASE.jar lrwxrwxrwx. 1 root root 30 Nov 4 18:00 spring-tx.jar -> ../spring-tx-5.0.4.RELEASE.jar But now there is no '4.4' directory. The engine spec still has: %global ovirt_dependencies ovirt-dependencies/4.4 despite being at 4.5 and despite also ovirt-dependencies being bumped to 4.5 a few days ago. Not sure what the exact plans are regarding this change - if it's just a bug, I'll let Martin/Artur fix (by perhaps adding a 4.5 directory with links and point the 4.5 engine there, or make it provide both - not sure what the plans are). I did not try to revert, anyway, because quite many other related patches were merged since, including the move to github/copr. For now, I pushed this simple workaround to OST [1]: # ln -s . /usr/share/java/ovirt-dependencies/4.4 # ls -l /usr/share/java/ovirt-dependencies/ total 4724 lrwxrwxrwx. 1 root root 1 Dec 27 08:28 4.4 -> . -rw-r--r--. 1 root root 360096 Dec 26 23:39 spring-aop.jar -rw-r--r--. 1 root root 654084 Dec 26 23:39 spring-beans.jar -rw-r--r--. 1 root root 1079129 Dec 26 23:39 spring-context.jar -rw-r--r--. 1 root root 1216476 Dec 26 23:39 spring-core.jar -rw-r--r--. 1 root root 263472 Dec 26 23:39 spring-expression.jar -rw-r--r--. 1 root root 1452 Dec 26 23:39 spring-instrument.jar -rw-r--r--. 1 root root 15898 Dec 26 23:39 spring-jcl.jar -rw-r--r--. 1 root root 391565 Dec 26 23:39 spring-jdbc.jar -rw-r--r--. 1 root root 590394 Dec 26 23:39 spring-test.jar -rw-r--r--. 1 root root 249289 Dec 26 23:39 spring-tx.jar With this: # ls -l /usr/share/ovirt-engine/modules/common/org/springframework/main/ total 4 -rw-r--r--. 1 root root 737 Dec 23 15:29 module.xml lrwxrwxrwx. 1 root root 53 Dec 23 15:42 spring-aop.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-aop.jar lrwxrwxrwx. 1 root root 53 Dec 23 15:42 spring-asm.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-asm.jar lrwxrwxrwx. 1 root root 55 Dec 23 15:42 spring-beans.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-beans.jar lrwxrwxrwx. 1 root root 57 Dec 23 15:42 spring-context.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-context.jar lrwxrwxrwx. 1 root root 54 Dec 23 15:42 spring-core.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-core.jar lrwxrwxrwx. 1 root root 60 Dec 23 15:42 spring-expression.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-expression.jar lrwxrwxrwx. 1 root root 60 Dec 23 15:42 spring-instrument.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-instrument.jar lrwxrwxrwx. 1 root root 54 Dec 23 15:42 spring-jdbc.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-jdbc.jar lrwxrwxrwx. 1 root root 52 Dec 23 15:42 spring-tx.jar -> /usr/share/java/ovirt-dependencies/4.4/spring-tx.jar This unbreaks all the links despite one: spring-asm.jar is still missing. No idea why. It's missing also on my slightly older setup from ovirt-dependencies commit ca19602 (and engine 3326bfea06). I guess it might be unrelated. I am testing [1] now locally. It already passed test_add_cpu_profile so seems promising. Best regards, [1] https://gerrit.ovirt.org/c/ovirt-system-tests/+/118150 > [3] https://gerrit.ovirt.org/c/ovirt-dependencies/+/114654 > [4] https://gerrit.ovirt.org/c/ovirt-dependencies/+/117459 > _______________________________________________ > Devel mailing list -- devel@ovirt.org > To unsubscribe send an email to devel-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MON3B5NSSXBL2BFSLYYIMOTKUR2CK732/ -- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/Z5ATDX5D7M4E35J5Z7DJI4O36VKEM75L/