Hi,
I'm unable to reproduce that issue outside OST, following scenarios worked
without any issues:
Scenario 1
1. Make sure that selinux-policy-*3.13.1-229.el7_6.9 is not installed
2. Install and configure ovirt-engine 4.2.8
3. Login to webadmin - everything works fine
4. Update to selinux-policy-*3.13.1-229.el7_6.9
5. Login to webadmin - everything works fine
6. Try to restart ovirt-engine and rh-postgresql95-postgresql services
7. Login to webadmin - everything works fine
8. Upgrade all other available packages
9. Login to webadmin - everything works fine
10. Reboot the machine
11. Login to webadmin - everything works fine
Senario 2
1. Update CentOS to latest version and make sure that
selinux-policy-*3.13.1-229.el7_6.9 is installed
2. Install and configure ovirt-engine 4.2.8
3. Login to webadmin - everything works fine
So continuing the investigation, but so far it seems to me related only to
OST
Martin
On Mon, Feb 18, 2019 at 7:39 AM Eitan Raviv <era...@redhat.com> wrote:
> Just to add some coal to the fire, here are my findings for failures of
> the 4.2 OST network suite:
>
> Following the selinux update [0], engine setup fails because what looks
> like failure of engine to communicate with postgresql.
> In [1]:
>
> Feb 16 19:26:55 lago-network-suite-4-2-engine systemd: Starting PostgreSQL
> database server...
> Feb 16 19:26:55 lago-network-suite-4-2-engine postgresql-ctl: postgres cannot
> access the server configuration file
> "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission
> denied
> Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: pg_ctl: could
> not start server
> Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: Examine the log
> output.
> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd:
> rh-postgresql95-postgresql.service: control process exited, code=exited
> status=1
> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Failed to start
> PostgreSQL database server.
> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Unit
> rh-postgresql95-postgresql.service entered failed state.
> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd:
> rh-postgresql95-postgresql.service failed.
>
> and in [2] there are selinux access denials for pg_ctl to read the
> postgres.conf file:
>
> type=AVC msg=audit(1550363215.978:1067): avc: denied { read } for pid=8648
> comm="pg_ctl" name="postgresql.conf" dev="vda4" ino=888710
> scontext=system_u:system_r:postgresql_t:s0
> tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0
> type=SYSCALL msg=audit(1550363215.978:1067): arch=c000003e syscall=2
> success=no exit=-13 a0=7ffe611ff730 a1=0 a2=1b6 a3=24 items=0 ppid=1 pid=8648
> auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26
> fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl"
> exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl"
> subj=system_u:system_r:postgresql_t:s0 key=(null)
> type=PROCTITLE msg=audit(1550363215.978:1067):
> proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730
> type=AVC msg=audit(1550363215.978:1068): avc: denied { getattr } for
> pid=8648 comm="pg_ctl"
> path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/PG_VERSION" dev="vda4"
> ino=888709 scontext=system_u:system_r:postgresql_t:s0
> tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0
> type=SYSCALL msg=audit(1550363215.978:1068): arch=c000003e syscall=4
> success=no exit=-13 a0=60a640 a1=7ffe611ffa50 a2=7ffe611ffa50
> a3=2f62696c2f35396c items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26
> euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295
> comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl"
> subj=system_u:system_r:postgresql_t:s0 key=(null)
> type=PROCTITLE msg=audit(1550363215.978:1068):
> proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730
> type=AVC msg=audit(1550363215.994:1069): avc: denied { getattr } for
> pid=8654 comm="postgres"
> path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="vda4"
> ino=888710 scontext=system_u:system_r:postgresql_t:s0
> tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0
> type=SYSCALL msg=audit(1550363215.994:1069): arch=c000003e syscall=4
> success=no exit=-13 a0=1d862b0 a1=7fff91968710 a2=7fff91968710
> a3=2f62696c2f35396c items=0 ppid=8648 pid=8654 auid=4294967295 uid=26 gid=26
> euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295
> comm="postgres" exe="/opt/rh/rh-postgresql95/root/usr/bin/postgres"
> subj=system_u:system_r:postgresql_t:s0 key=(null)
>
> whereas in [3] - the build just before the selinux package update, these
> errors did not occur.
>
> Looks like alongside enabling selinux a policy update is required.
>
> thanks
>
>
> [0] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/900/
> [1]
> https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artifact/exported-artifacts/pre-tests/lago-network-suite-4-2-engine/_var_log/messages/*view*/
> [2]
> https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artifact/exported-artifacts/pre-tests/lago-network-suite-4-2-engine/_var_log/audit/audit.log/*view*/
> [3]
> https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/899/artifact/exported-artifacts/pre-tests/lago-network-suite-4-2-engine/_var_log/messages/*view*/
>
>
> On Sun, Feb 17, 2019 at 11:16 PM Dafna Ron <d...@redhat.com> wrote:
>
>> I think this is a regression causing rh-postgress to fail to start on
>> selinux conf.
>> the issue is probably with the selinux packages
>>
>> I ran lago locally to debug and ssh-ed to the vms and this is the output
>> from the processes start:
>>
>> Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine
>> postfix/postdrop[9028]: warning: unable to look up public/pickup: No such
>> file or directory
>> Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine
>> postfix/postdrop[9029]: warning: unable to look up public/pickup: No such
>> file or directory
>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine
>> polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610
>> (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], ob
>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine systemd[1]:
>> Starting PostgreSQL database server...
>> -- Subject: Unit rh-postgresql95-postgresql.service has begun start-up
>> -- Defined-By: systemd
>> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>> --
>> -- Unit rh-postgresql95-postgresql.service has begun starting up.
>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine
>> postgresql-ctl[9041]: postgres cannot access the server configuration file
>> "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission d
>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine
>> postgresql-ctl[9041]: pg_ctl: could not start server
>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine
>> postgresql-ctl[9041]: Examine the log output.
>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]:
>> rh-postgresql95-postgresql.service: control process exited, code=exited
>> status=1
>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]:
>> Failed to start PostgreSQL database server.
>> -- Subject: Unit rh-postgresql95-postgresql.service has failed
>> -- Defined-By: systemd
>> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>> --
>> -- Unit rh-postgresql95-postgresql.service has failed.
>> --
>> -- The result is failed.
>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]:
>> Unit rh-postgresql95-postgresql.service entered failed state.
>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]:
>> rh-postgresql95-postgresql.service failed.
>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine
>> polkitd[2720]: Unregistered Authentication Agent for
>> unix-process:9033:93610 (system bus name :1.160, object path
>> /org/freedesktop/PolicyKit1/Authent
>> Feb 17 16:03:01 lago-upgrade-from-release-suite-master-engine systemd[1]:
>> Started Session 51 of user root.
>> -- Subject: Unit session-51.scope has finished start-up
>> -- Defined-By: systemd
>>
>>
>>
>> Secure log:
>>
>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine
>> polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610
>> (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback],
>> object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale
>> en_US.UTF-8)
>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine
>> polkitd[2720]: Unregistered Authentication Agent for
>> unix-process:9033:93610 (system bus name :1.160, object path
>> /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
>> (disconnected from bus)
>>
>> after setenforce:
>>
>> root@lago-upgrade-from-release-suite-master-engine ~]# setenforce 0
>> [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl start
>> rh-postgresql95-postgresql.service
>> [root@lago-upgrade-from-release-suite-master-engine ~]#
>> [root@lago-upgrade-from-release-suite-master-engine ~]#
>> [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl status
>> rh-postgresql95-postgresql.service
>> ● rh-postgresql95-postgresql.service - PostgreSQL database server
>> Loaded: loaded
>> (/usr/lib/systemd/system/rh-postgresql95-postgresql.service; disabled;
>> vendor preset: disabled)
>> Active: active (running) since Sun 2019-02-17 16:08:18 EST; 7s ago
>> Process: 9137
>> ExecStart=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-ctl start -D
>> ${PGDATA} -s -w -t ${PGSTARTTIMEOUT} (code=exited, status=0/SUCCESS)
>> Process: 9134
>> ExecStartPre=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-check-db-dir
>> %N (code=exited, status=0/SUCCESS)
>> Main PID: 9143 (postgres)
>> CGroup: /system.slice/rh-postgresql95-postgresql.service
>> ├─9143 /opt/rh/rh-postgresql95/root/usr/bin/postgres -D
>> /var/opt/rh/rh-postgresql95/lib/pgsql/data
>> ├─9144 postgres: logger process
>> ├─9146 postgres: checkpointer process
>> ├─9147 postgres: writer process
>> ├─9148 postgres: wal writer process
>> ├─9149 postgres: autovacuum launcher process
>> └─9150 postgres: stats collector process
>>
>> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine systemd[1]:
>> Starting PostgreSQL database server...
>> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine
>> postgresql-ctl[9137]: LOG: redirecting log output to logging collector
>> process
>> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine
>> postgresql-ctl[9137]: HINT: Future log output will appear in directory
>> "pg_log".
>> Feb 17 16:08:18 lago-upgrade-from-release-suite-master-engine systemd[1]:
>> Started PostgreSQL database server.
>> [root@lago-upgrade-from-release-suite-master-engine ~]#
>>
>> Not sure who deals with this configuration but this is a blocker as
>> upgrade from release is failing for both ovirt-engine and vdsm.
>>
>> Thanks,
>> Dafna
>>
>>
>> On Sun, Feb 17, 2019 at 10:55 AM Galit Rosenthal <grose...@redhat.com>
>> wrote:
>>
>>> Thanks Greg
>>>
>>> I will check this
>>>
>>>
>>> On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta <gsher...@redhat.com>
>>> wrote:
>>>
>>>> Is there any way you can run
>>>> "systemctl status rh-postgresql95-postgresql.service" and "journalctl
>>>> -xe"
>>>> like it suggests?
>>>> The logs below don't give any indication why it failed to start, afaict.
>>>>
>>>> On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal <grose...@redhat.com>
>>>> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I receive this error message both in CQ and check_patch:
>>>>>
>>>>> 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd
>>>>> systemd.state:130 starting service rh-postgresql95-postgresql
>>>>> 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd
>>>>> plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start',
>>>>> 'rh-postgresql95-postgresql.service'), executable='None', cwd='None',
>>>>> env=None
>>>>> 2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd
>>>>> plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start',
>>>>> 'rh-postgresql95-postgresql.service'), rc=1
>>>>> 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd
>>>>> plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start',
>>>>> 'rh-postgresql95-postgresql.service') stdout:
>>>>>
>>>>>
>>>>> 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd
>>>>> plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start',
>>>>> 'rh-postgresql95-postgresql.service') stderr:
>>>>> Job for rh-postgresql95-postgresql.service failed because the control
>>>>> process exited with error code. See "systemctl status
>>>>> rh-postgresql95-postgresql.service" and "journalctl -xe" for details.
>>>>>
>>>>> 2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction
>>>>> transaction.abort:119 aborting 'File transaction for
>>>>> '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf''
>>>>> 2019-02-16 16:28:07,916-0500 DEBUG otopi.context
>>>>> context._executeMethod:143 method exception
>>>>> Traceback (most recent call last):
>>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in
>>>>> _executeMethod
>>>>> method['method']()
>>>>> File
>>>>> "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py",
>>>>> line 201, in _misc
>>>>> self._provisioning.provision()
>>>>> File
>>>>> "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py",
>>>>> line 498, in provision
>>>>> self.restartPG()
>>>>> File
>>>>> "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py",
>>>>> line 399, in restartPG
>>>>> state=state,
>>>>> File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in
>>>>> state
>>>>> service=name,
>>>>> RuntimeError: Failed to start service 'rh-postgresql95-postgresql'
>>>>> 2019-02-16 16:28:07,918-0500 ERROR otopi.context
>>>>> context._executeMethod:152 Failed to execute stage 'Misc configuration':
>>>>> Failed to start service 'rh-postgresql95-postgresql'
>>>>> 2019-02-16 16:28:07,958-0500 DEBUG
>>>>> otopi.plugins.otopi.debug.debug_failure.debug_failure
>>>>> debug_failure._notification:100 tcp connections:
>>>>> id uid local foreign state pid exe
>>>>>
>>>>>
>>>>> What can cause it?
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Galit
>>>>>
>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change-queue-tester/12916/testReport/junit/(root)/001_initialize_engine/running_tests___upgrade_from_release_suite_el7_x86_64___test_initialize_engine/
>>>>>
>>>>>
>>>>> https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_standard-check-patch/detail/ovirt-system-tests_standard-check-patch/3207/pipeline/101
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Galit
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> GALIT ROSENTHAL
>>>>>
>>>>> SOFTWARE ENGINEER
>>>>>
>>>>> Red Hat
>>>>>
>>>>> <https://www.redhat.com/>
>>>>>
>>>>> ga...@gmail.com T: 972-9-7692230
>>>>> <https://red.ht/sig>
>>>>> _______________________________________________
>>>>> Devel mailing list -- devel@ovirt.org
>>>>> To unsubscribe send an email to devel-le...@ovirt.org
>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>>> oVirt Code of Conduct:
>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>> List Archives:
>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QNDG65M6UPEXTCT3HXORRTZ67RVXH653/
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> GREG SHEREMETA
>>>>
>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
>>>>
>>>> Red Hat NA
>>>>
>>>> <https://www.redhat.com/>
>>>>
>>>> gsher...@redhat.com IRC: gshereme
>>>> <https://red.ht/sig>
>>>>
>>>
>>>
>>> --
>>>
>>> GALIT ROSENTHAL
>>>
>>> SOFTWARE ENGINEER
>>>
>>> Red Hat
>>>
>>> <https://www.redhat.com/>
>>>
>>> ga...@gmail.com T: 972-9-7692230
>>> <https://red.ht/sig>
>>> _______________________________________________
>>> Devel mailing list -- devel@ovirt.org
>>> To unsubscribe send an email to devel-le...@ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YROV4PLNBTOWKYMT2EL25CN3C26HOU2R/
>>>
>> _______________________________________________
>> Devel mailing list -- devel@ovirt.org
>> To unsubscribe send an email to devel-le...@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CSNQENF4J6ZQJGS5T4QQMRRBDGZG6J4L/
>>
>
--
Martin Perina
Associate Manager, Software Engineering
Red Hat Czech s.r.o.
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UWIPGYFWNCG5H2NLN5CP4HW35YLGGAC6/
