On 11/30/2017 03:03 PM, Yedidyah Bar David wrote: > On Wed, Nov 29, 2017 at 4:57 PM, Yedidyah Bar David <d...@redhat.com> wrote: >> On Wed, Nov 29, 2017 at 3:56 PM, Dafna Ron <d...@redhat.com> wrote: >>> we had a failure on 002_bootstrap.verify_add_hosts but the error is on >>> imageio >>> >>> I looked at the host log that Nir added and I can only see that the >>> address is in use which seems to be the same issue we have in initialize >>> engine. >>> >>> >>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4205/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-host-0/_var_log/ovirt-imageio-daemon/ >>> >>> I cannot see anything in host-deploy. >>> Didi, would we be able to see anything here? >> >> Sorry, seems like my plugin is not enough. Will have a look. > Now merged an updated plugin, should hopefully pass changequeue soon. > Let's see what happens next time a service fails. > Search engine-setup/host-deploy logs for 'tcp connections'. > > Best regards, thank you :) will update once we have another failure. >>> >>> Thanks, >>> Dafna >>> >>> >>> >>> On 11/29/2017 11:03 AM, Yedidyah Bar David wrote: >>> >>> On Wed, Nov 29, 2017 at 1:00 PM, Dafna Ron <d...@redhat.com> wrote: >>>> this is the plugin info from steup log but I don't see anything more than >>>> we have seen except a timeout. >>>> >>>> https://pastebin.com/QVtNRNWV >>>> >>>> >>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20171128123116-mmjen3.log >>>> >>>> Didi, is there anywhere else I should look? >>> >>> Sadly, as already replied, not yet. Hopefully next time... >>> >>>> >>>> >>>> On 11/29/2017 10:18 AM, Nir Soffer wrote: >>>> >>>> Do we have more info from Didi's debug plugin now? >>>> >>>> On Wed, Nov 29, 2017 at 12:07 PM Dafna Ron <d...@redhat.com> wrote: >>>>> Hi, >>>>> >>>>> We have failed cq with ovirt-imageio failing to start on upgrade suite. >>>>> I can still only see errors in the messages log. >>>>> >>>>> I'm writing the reported patch but I don't think it has anything to do >>>>> with this issue. >>>>> >>>>> Link and headline of suspected patches: >>>>> >>>>> restapi: Enable update to no default network provider of cluster - >>>>> https://gerrit.ovirt.org/#/c/84814/ >>>>> >>>>> Link to Job: >>>>> >>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/ >>>>> >>>>> Link to all logs: >>>>> >>>>> >>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/ >>>>> >>>>> >>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/testReport/junit/(root)/001_initialize_engine/test_initialize_engine/ >>>>> >>>>> >>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4194/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-001_initialize_engine.py/lago-upgrade-from-release-suite-master-engine/_var_log/messages/*view*/ >>>>> >>>>> (Relevant) error snippet from the log: >>>>> >>>>> <error> >>>>> >>>>> From messages log >>>>> >>>>> >>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Started oVirt Engine. >>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Reloading. >>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Configuration file /usr/lib/systemd/system/ebtables.service is marked >>>>> executable. Please remove executable permission bits. Proceeding anyway. >>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Starting oVirt Engine Data Warehouse... >>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Started oVirt Engine Data Warehouse. >>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Reloading. >>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Configuration file /usr/lib/systemd/system/ebtables.service is marked >>>>> executable. Please remove executable permission bits. Proceeding anyway. >>>>> Nov 28 12:32:13 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Starting oVirt ImageIO Proxy... >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: Traceback (most recent call last): >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in >>>>> <module> >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: status = image_proxy.main(args, config) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File >>>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", >>>>> line >>>>> 21, in main >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: image_server.start(config) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File >>>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, >>>>> in start >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: WSGIRequestHandler) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line >>>>> 419, >>>>> in __init__ >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: self.server_bind() >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", >>>>> line 48, in server_bind >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line >>>>> 108, in server_bind >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line >>>>> 430, >>>>> in server_bind >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in >>>>> meth >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> ovirt-imageio-proxy.service: main process exited, code=exited, >>>>> status=1/FAILURE >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Failed to start oVirt ImageIO Proxy. >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Unit ovirt-imageio-proxy.service entered failed state. >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> ovirt-imageio-proxy.service failed. >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart. >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Starting oVirt ImageIO Proxy... >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: Traceback (most recent call last): >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/bin/ovirt-imageio-proxy", line 85, in >>>>> <module> >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: status = image_proxy.main(args, config) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File >>>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/image_proxy.py", >>>>> line >>>>> 21, in main >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: image_server.start(config) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File >>>>> "/usr/lib/python2.7/site-packages/ovirt_imageio_proxy/server.py", line 45, >>>>> in start >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: WSGIRequestHandler) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line >>>>> 419, >>>>> in __init__ >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: self.server_bind() >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/wsgiref/simple_server.py", >>>>> line 48, in server_bind >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: HTTPServer.server_bind(self) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/BaseHTTPServer.py", line >>>>> 108, in server_bind >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: SocketServer.TCPServer.server_bind(self) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/SocketServer.py", line >>>>> 430, >>>>> in server_bind >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: self.socket.bind(self.server_address) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: File "/usr/lib64/python2.7/socket.py", line 224, in >>>>> meth >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: return getattr(self._sock,name)(*args) >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine >>>>> ovirt-imageio-proxy: socket.error: [Errno 98] Address already in use >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> ovirt-imageio-proxy.service: main process exited, code=exited, >>>>> status=1/FAILURE >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Failed to start oVirt ImageIO Proxy. >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> Unit ovirt-imageio-proxy.service entered failed state. >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> ovirt-imageio-proxy.service failed. >>>>> Nov 28 12:32:14 lago-upgrade-from-release-suite-master-engine systemd: >>>>> ovirt-imageio-proxy.service holdoff time over, scheduling restart. >>>>> >>>>> </error> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >>> -- >>> Didi >>> >>> >> >> >> -- >> Didi > >
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel