When you say you run the playbook again and got the same error, which error do you mean? The "No such file or directory: '/var/lib/pulp/assets/rest_framework/js/jquery-3.4.1.min.js'" one?
David On Thu, Apr 15, 2021 at 4:43 AM Eric VS <vs.e...@gmail.com> wrote: > Hey Matthias, > > I ran the pulpcore-manager collectstatic with success, then the playbook > again but got the same error. This is the full list of commands for > pulpcore-manager, any others I would need to run? > > Last login: Thu Apr 15 16:23:00 CST 2021 on pts/0 > [root@tw-stg-man01 ~]# export PULP_SETTINGS=/etc/pulp/settings.py && > /usr/local/lib/pulp/bin/pulpcore-manager > > Type 'pulpcore-manager help <subcommand>' for help on a specific > subcommand. > > Available subcommands: > > [app] > handle-artifact-checksums > reset-admin-password > stage-profile-summary > > [auth] > changepassword > createsuperuser > > [contenttypes] > remove_stale_contenttypes > > [django] > check > compilemessages > createcachetable > dbshell > diffsettings > dumpdata > flush > inspectdb > loaddata > makemessages > makemigrations > migrate > sendtestemail > shell > showmigrations > sqlflush > sqlmigrate > sqlsequencereset > squashmigrations > startapp > startproject > test > testserver > > [drf_spectacular] > spectacular > > [guardian] > clean_orphan_obj_perms > > [rest_framework] > generateschema > > [sessions] > clearsessions > > [staticfiles] > collectstatic > findstatic > runserver > > *Eric Van Steenbergen* > > *E-mail: vs.e...@gmail.com <vs.e...@gmail.com>* > *Skype: ericvs2014* > *LinkedIn: Eric Van Steenbergen > <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* > > > On Thu, Apr 15, 2021 at 3:13 AM Matthias Dellweg <mdell...@redhat.com> > wrote: > >> Yeah, i can see how that happened. There may be some triggers from the >> previous run that aren't triggered now, because "nothing changed" according >> to ansible. >> This one should be: pulpcore-manager collect-static (or very similar) >> When you just run pulpcore-manager you get a list where you should be >> able to spot that subcommand. >> >> On Thu, Apr 15, 2021 at 10:06 AM Eric VS <vs.e...@gmail.com> wrote: >> >>> Hi Matthias, >>> >>> I did try to run that command but thought it was failing due to the last >>> error. >>> Unfortunately, when I just tried running the playbook again I ran into >>> the next error :-) >>> >>> TASK [pulp.pulp_installer.pulp_health_check : Checking Pulp services] >>> *********************************************************************************************************************************************************** >>> skipping: [localhost] => (item={'key': 'pulpcore-worker@2.service', >>> 'value': {'status': 'unknown', 'source': 'systemd', 'state': 'running', >>> 'name': 'pulpcore-worker@2.service'}}) >>> skipping: [localhost] => (item={'key': >>> 'pulpcore-resource-manager.service', 'value': {'status': 'enabled', >>> 'source': 'systemd', 'state': 'running', 'name': >>> 'pulpcore-resource-manager.service'}}) >>> skipping: [localhost] => (item={'key': 'pulpcore-worker@.service', >>> 'value': {'status': 'enabled', 'source': 'systemd', 'state': 'unknown', >>> 'name': 'pulpcore-worker@.service'}}) >>> skipping: [localhost] => (item={'key': 'pulpcore-worker@1.service', >>> 'value': {'status': 'unknown', 'source': 'systemd', 'state': 'running', >>> 'name': 'pulpcore-worker@1.service'}}) >>> failed: [localhost] (item={'key': 'pulpcore-api.service', 'value': >>> {'status': 'enabled', 'source': 'systemd', 'state': 'stopped', 'name': >>> 'pulpcore-api.service'}}) => {"ansible_loop_var": "item", "changed": false, >>> "item": {"key": "pulpcore-api.service", "value": {"name": >>> "pulpcore-api.service", "source": "systemd", "state": "stopped", "status": >>> "enabled"}}, "msg": "pulpcore-api.service state: stopped"} >>> skipping: [localhost] => (item={'key': 'pulpcore-content.service', >>> 'value': {'status': 'enabled', 'source': 'systemd', 'state': 'running', >>> 'name': 'pulpcore-content.service'}}) >>> >>> PLAY RECAP >>> ********************************************************************************************************************************************************************************************************************** >>> localhost : ok=115 changed=19 unreachable=0 >>> failed=1 skipped=50 rescued=0 ignored=0 >>> >>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) sudo systemctl start >>> pulpcore-api >>> >>> 2 ↵ >>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) ^start^status >>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) sudo systemctl status >>> pulpcore-api >>> ● pulpcore-api.service - Pulp WSGI Server >>> Loaded: loaded (/usr/lib/systemd/system/pulpcore-api.service; >>> enabled; vendor preset: disabled) >>> Active: activating (auto-restart) (Result: exit-code) since Thu >>> 2021-04-15 16:03:12 CST; 1s ago >>> Process: 4875 ExecStart=/usr/local/lib/pulp/bin/gunicorn >>> pulpcore.app.wsgi:application --bind 127.0.0.1:24817 --workers 1 >>> --timeout 90 --access-logfile - --access-logformat pulp >>> [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s >>> "%(f)s" "%(a)s" (code=exited, status=3) >>> Main PID: 4875 (code=exited, status=3) >>> >>> Apr 15 16:03:12 tw-stg-man01.stg01.activeinfra.net systemd[1]: Unit >>> pulpcore-api.service entered failed state. >>> Apr 15 16:03:12 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>> pulpcore-api.service failed. >>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) sudo journalctl -u >>> pulpcore-api >>> >>> 3 ↵ >>> -- Logs begin at Mon 2020-07-20 11:44:39 CST, end at Thu 2021-04-15 >>> 16:03:38 CST. -- >>> Aug 05 10:24:09 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>> Started Pulp WSGI Server. >>> Aug 05 10:24:09 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>> [2020-08-05 10:24:09 +0800] [815] [INFO] Starting gunicorn 20.0.4 >>> Aug 05 10:24:09 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>> [2020-08-05 10:24:09 +0800] [815] [INFO] Listening at: >>> http://127.0.0.1:24817 (815) >>> Aug 05 10:24:09 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>> [2020-08-05 10:24:09 +0800] [815] [INFO] Using worker: sync >>> Aug 05 10:24:09 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>> [2020-08-05 10:24:09 +0800] [923] [INFO] Booting worker with pid: 923 >>> Aug 05 10:24:27 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>> Stopping Pulp WSGI Server... >>> Aug 05 10:24:27 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>> [2020-08-05 10:24:27 +0800] [815] [INFO] Handling signal: term >>> Aug 05 10:24:27 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>> [2020-08-05 02:24:27 +0000] [923] [INFO] Worker exiting (pid: 923) >>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net gunicorn[815]: >>> [2020-08-05 10:24:28 +0800] [815] [INFO] Shutting down: Master >>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>> Stopped Pulp WSGI Server. >>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>> Started Pulp WSGI Server. >>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net gunicorn[3794]: >>> [2020-08-05 10:24:28 +0800] [3794] [INFO] Starting gunicorn 20.0.4 >>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net gunicorn[3794]: >>> [2020-08-05 10:24:28 +0800] [3794] [INFO] Listening at: >>> http://127.0.0.1:24817 (3794) >>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net gunicorn[3794]: >>> [2020-08-05 10:24:28 +0800] [3794] [INFO] Using worker: sync >>> Aug 05 10:24:28 tw-stg-man01.app-stg01.activeinfra.net gunicorn[3794]: >>> [2020-08-05 10:24:28 +0800] [3873] [INFO] Booting worker with pid: 3873 >>> Aug 05 11:16:03 tw-stg-man01.app-stg01.activeinfra.net gunicorn[3794]: >>> [2020-08-05 11:16:03 +0800] [3794] [INFO] Handling signal: term >>> Aug 05 11:16:03 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>> Stopping Pulp WSGI Server... >>> Aug 05 11:16:03 tw-stg-man01.app-stg01.activeinfra.net gunicorn[3794]: >>> [2020-08-05 03:16:03 +0000] [3873] [INFO] Worker exiting (pid: 3873) >>> Aug 05 11:16:03 tw-stg-man01.app-stg01.activeinfra.net gunicorn[3794]: >>> [2020-08-05 11:16:03 +0800] [3794] [INFO] Shutting down: Master >>> Aug 05 11:16:03 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>> Stopped Pulp WSGI Server. >>> -- Reboot -- >>> Aug 05 11:19:03 tw-stg-man01.app-stg01.activeinfra.net systemd[1]: >>> Started Pulp WSGI Server. >>> Aug 05 11:19:05 tw-stg-man01.app-stg01.activeinfra.net gunicorn[1858]: >>> [2020-08-05 11:19:05 +0800] [1858] [INFO] Starting gunicorn 20.0.4 >>> Aug 05 11:19:05 tw-stg-man01.app-stg01.activeinfra.net gunicorn[1858]: >>> [2020-08-05 11:19:05 +0800] [1858] [INFO] Listening at: >>> http://127.0.0.1:24817 (1858) >>> Aug 05 11:19:05 tw-stg-man01.app-stg01.activeinfra.net gunicorn[1858]: >>> [2020-08-05 11:19:05 +0800] [1858] [INFO] Using worker: sync >>> Aug 05 11:19:05 tw-stg-man01.app-stg01.activeinfra.net gunicorn[1858]: >>> [2020-08-05 11:19:05 +0800] [2329] [INFO] Booting worker with pid: 2329 >>> Aug 05 11:38:18 tw-stg-man01.app-stg01.activeinfra.net gunicorn[1858]: >>> pulp: django.request:WARNING: Not Found: / >>> Aug 05 11:38:18 tw-stg-man01.app-stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - - [05/Aug/2020:03:38:18 +0000] "GET / HTTP/1.0" 404 77 "-" >>> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 >>> Firefox/81.0 >>> Aug 05 11:38:18 tw-stg-man01.app-stg01.activeinfra.net gunicorn[1858]: >>> pulp: django.request:WARNING: Not Found: /favicon.ico >>> Aug 05 11:38:18 tw-stg-man01.app-stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - - [05/Aug/2020:03:38:18 +0000] "GET /favicon.ico HTTP/1.0" 404 >>> 77 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) >>> Gecko/20100101 F >>> Aug 07 13:40:58 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - - [07/Aug/2020:05:40:58 +0000] "GET /pulp/api/v3/status/ >>> HTTP/1.0" 200 1129 "-" "curl/7.29.0" >>> Aug 07 13:41:07 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - - [07/Aug/2020:05:41:07 +0000] "GET /pulp/api/v3/status/ >>> HTTP/1.0" 200 1129 "-" "curl/7.29.0" >>> Aug 07 13:41:43 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - - [07/Aug/2020:05:41:43 +0000] "GET /pulp/api/v3/status/ >>> HTTP/1.0" 200 1129 "-" "curl/7.29.0" >>> Aug 07 13:44:48 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - - [07/Aug/2020:05:44:48 +0000] "GET /pulp/api/v3/status/ >>> HTTP/1.0" 200 1129 "-" "curl/7.29.0" >>> Aug 07 13:44:59 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - - [07/Aug/2020:05:44:59 +0000] "GET /pulp/api/v3/status/ >>> HTTP/1.0" 200 1129 "-" "curl/7.29.0" >>> Aug 07 13:48:11 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - - [07/Aug/2020:05:48:11 +0000] "GET /pulp/api/v3/status/ >>> HTTP/1.0" 200 1129 "-" "HTTPie/0.9.4" >>> Aug 10 10:34:07 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - - [10/Aug/2020:02:34:07 +0000] "GET /pulp/api/v3/status/ >>> HTTP/1.0" 200 1129 "-" "HTTPie/0.9.4" >>> Aug 10 12:55:34 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - - [10/Aug/2020:04:55:34 +0000] "GET /pulp/api/v3/status/ >>> HTTP/1.0" 200 1129 "-" "HTTPie/0.9.4" >>> Aug 10 13:01:47 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> pulp: django.request:WARNING: Not Found: / >>> Aug 10 13:01:47 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:01:47 +0000] "GET / HTTP/1.1" 404 77 "-" >>> "HTTPie/0.9.4" >>> Aug 10 13:02:01 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> pulp: django.request:WARNING: Not Found: / >>> Aug 10 13:02:01 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:02:01 +0000] "GET / HTTP/1.1" 404 77 "-" >>> "HTTPie/0.9.4" >>> Aug 10 13:03:16 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:16 +0000] "POST >>> /pulp/api/v3/repositories/rpm/rpm/ HTTP/1.1" 201 441 "-" "HTTPie/0.9.4" >>> Aug 10 13:03:16 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:16 +0000] "GET >>> /pulp/api/v3/repositories/rpm/rpm/46d2f3e3-904e-448c-ae36-8d0662788f8e/ >>> HTTP/1.1" 200 441 "-" "HTTPie/0.9. >>> Aug 10 13:03:16 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:16 +0000] "POST >>> /pulp/api/v3/remotes/rpm/rpm/ HTTP/1.1" 201 462 "-" "HTTPie/0.9.4" >>> Aug 10 13:03:17 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:17 +0000] "GET >>> /pulp/api/v3/remotes/rpm/rpm/ HTTP/1.1" 200 514 "-" "HTTPie/0.9.4" >>> Aug 10 13:03:17 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:17 +0000] "GET >>> /pulp/api/v3/remotes/rpm/rpm/a4e0901e-5091-4a3e-ad6d-41f1f0c88d1e/ >>> HTTP/1.1" 200 462 "-" "HTTPie/0.9.4" >>> Aug 10 13:03:17 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:17 +0000] "POST >>> /pulp/api/v3/repositories/rpm/rpm/46d2f3e3-904e-448c-ae36-8d0662788f8e/sync/ >>> HTTP/1.1" 202 67 "-" "HTTPie >>> Aug 10 13:03:17 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:17 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 606 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:19 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:19 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1094 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:20 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:20 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1201 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:22 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:22 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1201 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:23 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:23 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1201 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:24 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:24 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1201 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:26 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:26 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1201 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:27 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:27 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1201 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:28 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:28 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1201 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:30 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:30 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1201 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:31 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:31 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1201 >>> "-" "HTTPie/0.9.4" >>> Aug 10 13:03:33 tw-stg-man01.stg01.activeinfra.net gunicorn[1858]: >>> 127.0.0.1 - admin [10/Aug/2020:05:03:33 +0000] "GET >>> /pulp/api/v3/tasks/169ca438-1c83-4af6-899c-ad228fc551bc/ HTTP/1.1" 200 1201 >>> "-" "HTTPie/0.9.4" >>> ...skipping... >>> Apr 15 16:03:34 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>> pulpcore-api.service: main process exited, code=exited, >>> status=3/NOTIMPLEMENTED >>> Apr 15 16:03:34 tw-stg-man01.stg01.activeinfra.net systemd[1]: Unit >>> pulpcore-api.service entered failed state. >>> Apr 15 16:03:34 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>> pulpcore-api.service failed. >>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>> pulpcore-api.service holdoff time over, scheduling restart. >>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net systemd[1]: Stopped >>> Pulp WSGI Server. >>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net systemd[1]: Started >>> Pulp WSGI Server. >>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> [2021-04-15 16:03:37 +0800] [4938] [INFO] Starting gunicorn 20.1.0 >>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> [2021-04-15 16:03:37 +0800] [4938] [INFO] Listening at: >>> http://127.0.0.1:24817 (4938) >>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> [2021-04-15 16:03:37 +0800] [4938] [INFO] Using worker: sync >>> Apr 15 16:03:37 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> [2021-04-15 16:03:37 +0800] [4941] [INFO] Booting worker with pid: 4941 >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> [2021-04-15 08:03:39 +0000] [4941] [ERROR] Exception in worker process >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> Traceback (most recent call last): >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/arbiter.py", >>> line 589, in spawn_worker >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> worker.init_process() >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/workers/base.py", >>> line 134, in init_process >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> self.load_wsgi() >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/workers/base.py", >>> line 146, in load_wsgi >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> self.wsgi = self.app.wsgi() >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/app/base.py", >>> line 67, in wsgi >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> self.callable = self.load() >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/app/wsgiapp.py", >>> line 58, in load >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> return self.load_wsgiapp() >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/app/wsgiapp.py", >>> line 48, in load_wsgiapp >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> return util.import_app(self.app_uri) >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/gunicorn/util.py", line >>> 359, in import_app >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: mod >>> = importlib.import_module(module) >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> return _bootstrap._gcd_import(name[level:], package, level) >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "<frozen importlib._bootstrap>", line 994, in _gcd_import >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "<frozen importlib._bootstrap>", line 971, in _find_and_load >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "<frozen importlib._bootstrap>", line 665, in _load_unlocked >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "<frozen importlib._bootstrap_external>", line 678, in exec_module >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/wsgi.py", >>> line 17, in <module> >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> application = get_wsgi_application() >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/wsgi.py", >>> line 13, in get_wsgi_application >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> return WSGIHandler() >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/wsgi.py", >>> line 135, in __init__ >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> self.load_middleware() >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py", >>> line 37, in load_middleware >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> mw_instance = middleware(handler) >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/middleware.py", >>> line 49, in __init__ >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> self.add_files(self.static_root, prefix=self.static_prefix) >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/base.py", >>> line 113, in add_files >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> self.update_files_dictionary(root, prefix) >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/base.py", >>> line 120, in update_files_dictionary >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> stat_cache = dict(scantree(root)) >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/base.py", >>> line 271, in scantree >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> yield from scantree(entry.path) >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/base.py", >>> line 271, in scantree >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> yield from scantree(entry.path) >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: File >>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/whitenoise/base.py", >>> line 273, in scantree >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> yield entry.path, entry.stat() >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> FileNotFoundError: [Errno 2] No such file or directory: >>> '/var/lib/pulp/assets/rest_framework/js/jquery-3.4.1.min.js' >>> Apr 15 16:03:39 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> [2021-04-15 08:03:39 +0000] [4941] [INFO] Worker exiting (pid: 4941) >>> Apr 15 16:03:40 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> [2021-04-15 16:03:40 +0800] [4938] [INFO] Shutting down: Master >>> Apr 15 16:03:40 tw-stg-man01.stg01.activeinfra.net gunicorn[4938]: >>> [2021-04-15 16:03:40 +0800] [4938] [INFO] Reason: Worker failed to boot. >>> Apr 15 16:03:40 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>> pulpcore-api.service: main process exited, code=exited, >>> status=3/NOTIMPLEMENTED >>> Apr 15 16:03:40 tw-stg-man01.stg01.activeinfra.net systemd[1]: Unit >>> pulpcore-api.service entered failed state. >>> Apr 15 16:03:40 tw-stg-man01.stg01.activeinfra.net systemd[1]: >>> pulpcore-api.service failed. >>> >>> Kind regards, >>> >>> *Eric Van Steenbergen* >>> >>> *E-mail: vs.e...@gmail.com <vs.e...@gmail.com>* >>> *Skype: ericvs2014* >>> *LinkedIn: Eric Van Steenbergen >>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>> >>> >>> On Thu, Apr 15, 2021 at 2:56 AM Matthias Dellweg <mdell...@redhat.com> >>> wrote: >>> >>>> >>>> There is the clue: Run 'pulpcore-manager handle-artifact-checksums' >>>> first to unset forbidden checksums. >>>> Pulp dropped to support md5 and sha1 by default. And there is a manual >>>> step that should bring the database in the right place. >>>> Try rerunning the ansible playbook afterwards. >>>> >>>> However i thought this cleanup routine should have been called >>>> automatically (as a migration) in the upgrade process. Can anyone spot why >>>> it didn't? >>>> >>>> On Thu, Apr 15, 2021 at 1:48 AM Eric VS <vs.e...@gmail.com> wrote: >>>> >>>>> Hey Matthias, >>>>> >>>>> The only plugin I am currently using, at least until I'm familiar >>>>> enough with Pulp, is the RPM plugin. That's our primary use case for now >>>>> to >>>>> make sure we are in control of what version of packages we are using >>>>> across >>>>> our environments and avoid discrepancies between servers. >>>>> >>>>> I have never added or removed any other plugin. After also upgrading >>>>> the Postgres ansible-galaxy role I don't get the version mismatch error >>>>> any >>>>> more but now get the below error and 'command to run'. Running the command >>>>> fails however also.... >>>>> >>>>> RUNNING HANDLER [pulp.pulp_installer.pulp_common : Collect static >>>>> content] >>>>> ****************************************************************************************************************************************************** >>>>> fatal: [localhost]: FAILED! => {"changed": true, "cmd": >>>>> ["/usr/local/lib/pulp/bin/pulpcore-manager", "collectstatic", "--noinput", >>>>> "--link"], "delta": "0:00:02.481870", "end": "2021-04-15 07:34:39.205489", >>>>> "msg": "non-zero return code", "rc": 1, "start": "2021-04-15 >>>>> 07:34:36.723619", "stderr": "Traceback (most recent call last):\n File >>>>> \"/usr/local/lib/pulp/bin/pulpcore-manager\", line 8, in <module>\n >>>>> sys.exit(manage())\n File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py\", >>>>> line 11, in manage\n execute_from_command_line(sys.argv)\n File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>> line 381, in execute_from_command_line\n utility.execute()\n File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>> line 357, in execute\n django.setup()\n File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/__init__.py\", >>>>> line 24, in setup\n apps.populate(settings.INSTALLED_APPS)\n File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/registry.py\", >>>>> line 114, in populate\n app_config.import_models()\n File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/config.py\", >>>>> line 211, in import_models\n self.models_module = >>>>> import_module(models_module_name)\n File >>>>> \"/usr/lib64/python3.6/importlib/__init__.py\", line 126, in >>>>> import_module\n return _bootstrap._gcd_import(name[level:], package, >>>>> level)\n File \"<frozen importlib._bootstrap>\", line 994, in >>>>> _gcd_import\n File \"<frozen importlib._bootstrap>\", line 971, in >>>>> _find_and_load\n File \"<frozen importlib._bootstrap>\", line 955, in >>>>> _find_and_load_unlocked\n File \"<frozen importlib._bootstrap>\", line >>>>> 665, in _load_unlocked\n File \"<frozen importlib._bootstrap_external>\", >>>>> line 678, in exec_module\n File \"<frozen importlib._bootstrap>\", line >>>>> 219, in _call_with_frames_removed\n File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/__init__.py\", >>>>> line 22, in <module>\n from .exporter import ( # noqa\n File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/exporter.py\", >>>>> line 15, in <module>\n from .task import CreatedResource, Task\n >>>>> >>>>> File\"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/task.py\", >>>>> line 14, in <module>\n from pulpcore.app.settings import WORKER_TTL\n >>>>> File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>> line 366, in <module>\n raise e\n File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>> line 344, in <module>\n >>>>> ).format(checksum)\ndjango.core.exceptions.ImproperlyConfigured: There >>>>> have been identified artifacts with forbidden checksum 'sha1'. Run >>>>> 'pulpcore-manager handle-artifact-checksums' first to unset forbidden >>>>> checksums.", "stderr_lines": ["Traceback (most recent call last):", " >>>>> File >>>>> \"/usr/local/lib/pulp/bin/pulpcore-manager\", line 8, in <module>", " >>>>> sys.exit(manage())", " File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py\", >>>>> line 11, in manage", " execute_from_command_line(sys.argv)", " File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>> line 381, in execute_from_command_line", " utility.execute()", " File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>> line 357, in execute", " django.setup()", " File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/__init__.py\", >>>>> line 24, in setup", " apps.populate(settings.INSTALLED_APPS)", " File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/registry.py\", >>>>> line 114, in populate", " app_config.import_models()", " File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/config.py\", >>>>> line 211, in import_models", " self.models_module = >>>>> import_module(models_module_name)", " File >>>>> \"/usr/lib64/python3.6/importlib/__init__.py\", line 126, in >>>>> import_module", " return _bootstrap._gcd_import(name[level:], package, >>>>> level)", " File \"<frozen importlib._bootstrap>\", line 994, in >>>>> _gcd_import", " File \"<frozen importlib._bootstrap>\", line 971, in >>>>> _find_and_load", " File \"<frozen importlib._bootstrap>\", line 955, in >>>>> _find_and_load_unlocked", " File \"<frozen importlib._bootstrap>\", line >>>>> 665, in _load_unlocked", " File \"<frozen >>>>> importlib._bootstrap_external>\", line 678, in exec_module", " File >>>>> \"<frozen importlib._bootstrap>\", line 219, in >>>>> _call_with_frames_removed", >>>>> " File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/__init__.py\", >>>>> line 22, in <module>", " from .exporter import ( # noqa", " File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/exporter.py\", >>>>> line 15, in <module>", " from .task import CreatedResource, Task", " >>>>> File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/task.py\", >>>>> line 14, in <module>", " from pulpcore.app.settings import WORKER_TTL", >>>>> " File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>> line 366, in <module>", " raise e", " File >>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>> line 344, in <module>", " ).format(checksum)", >>>>> "django.core.exceptions.ImproperlyConfigured: There have been identified >>>>> artifacts withforbidden checksum 'sha1'. Run 'pulpcore-manager >>>>> handle-artifact-checksums' first to unset forbidden checksums."], >>>>> "stdout": >>>>> "", "stdout_lines": []} >>>>> >>>>> RUNNING HANDLER [pulp.pulp_installer.pulp_common : Restart all Pulp >>>>> services] >>>>> *************************************************************************************************************************************************** >>>>> >>>>> NO MORE HOSTS LEFT >>>>> ************************************************************************************************************************************************************************************************************** >>>>> >>>>> PLAY RECAP >>>>> ********************************************************************************************************************************************************************************************************************** >>>>> localhost : ok=87 changed=3 unreachable=0 >>>>> failed=1 skipped=48 rescued=0 ignored=0 >>>>> >>>>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>>>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) export >>>>> PULP_SETTINGS=/etc/pulp/settings.py && >>>>> /usr/local/lib/pulp/bin/pulpcore-manager handle-artifact-checksums >>>>> 2 ↵ >>>>> Traceback (most recent call last): >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py", >>>>> line 283, in <module> >>>>> CONTENT_ORIGIN >>>>> NameError: name 'CONTENT_ORIGIN' is not defined >>>>> >>>>> During handling of the above exception, another exception occurred: >>>>> >>>>> Traceback (most recent call last): >>>>> File "/usr/local/lib/pulp/bin/pulpcore-manager", line 8, in <module> >>>>> sys.exit(manage()) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py", >>>>> line 11, in manage >>>>> execute_from_command_line(sys.argv) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py", >>>>> line 381, in execute_from_command_line >>>>> utility.execute() >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py", >>>>> line 357, in execute >>>>> django.setup() >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/__init__.py", >>>>> line 24, in setup >>>>> apps.populate(settings.INSTALLED_APPS) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/registry.py", >>>>> line 114, in populate >>>>> app_config.import_models() >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/apps/config.py", >>>>> line 211, in import_models >>>>> self.models_module = import_module(models_module_name) >>>>> File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in >>>>> import_module >>>>> return _bootstrap._gcd_import(name[level:], package, level) >>>>> File "<frozen importlib._bootstrap>", line 994, in _gcd_import >>>>> File "<frozen importlib._bootstrap>", line 971, in _find_and_load >>>>> File "<frozen importlib._bootstrap>", line 955, in >>>>> _find_and_load_unlocked >>>>> File "<frozen importlib._bootstrap>", line 665, in _load_unlocked >>>>> File "<frozen importlib._bootstrap_external>", line 678, in >>>>> exec_module >>>>> File "<frozen importlib._bootstrap>", line 219, in >>>>> _call_with_frames_removed >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/__init__.py", >>>>> line 22, in <module> >>>>> from .exporter import ( # noqa >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/exporter.py", >>>>> line 15, in <module> >>>>> from .task import CreatedResource, Task >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/models/task.py", >>>>> line 14, in <module> >>>>> from pulpcore.app.settings import WORKER_TTL >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py", >>>>> line 287, in <module> >>>>> "CONTENT_ORIGIN is a required setting but it was not configured. >>>>> This may be caused " >>>>> django.core.exceptions.ImproperlyConfigured: CONTENT_ORIGIN is a >>>>> required setting but it was not configured. This may be caused by invalid >>>>> read permissions of the settings file. Note that CONTENT_ORIGIN is set by >>>>> the installer automatically. >>>>> (ansible-venv) ╭─ericv@tw-stg-man01 ~ >>>>> ╰─$ (⎈ k8s01.stg01.activeinfra.net:linkerd) >>>>> >>>>> I finally had some time to dig deeper and played around with the >>>>> permissions on the file to no result, until I ran the pulpcore-manager >>>>> command as root. Then this popped up as error indicating there is some >>>>> integrity error: >>>>> >>>>> [root@tw-stg-man01 ~]# export PULP_SETTINGS=/etc/pulp/settings.py && >>>>> /usr/local/lib/pulp/bin/pulpcore-manager handle-artifact-checksums >>>>> Removing forbidden checksum sha1 from database >>>>> Traceback (most recent call last): >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", >>>>> line 84, in _execute >>>>> return self.cursor.execute(sql, params) >>>>> psycopg2.errors.NotNullViolation: null value in column "sha1" violates >>>>> not-null constraint >>>>> DETAIL: Failing row contains (1883479d-9e2d-45e1-971f-37246fc3c358, >>>>> 2020-08-13 05:50:33.543549+00, 2020-08-13 05:50:33.543572+00, >>>>> artifact/52/e229d511dca3c15b4250159cc7fd1685e8202cbb20e228bbd6ac..., >>>>> 286400, 51b3d1a3ff20a0f88ccb084eca86073c, null, >>>>> bd036226130d705e95bf4ad9acaa5d8d6d4df7460943062c08154bf1, >>>>> 52e229d511dca3c15b4250159cc7fd1685e8202cbb20e228bbd6ac0f16a27789, >>>>> 53964efdef715dd8f8ff96588e7f1ab184a3747d39bb3c9c28fc339659014274..., >>>>> 0b0d8a7d5d5816705016a5a59489f032fddc31afc438a482549091b730ca4f6b...). >>>>> >>>>> >>>>> The above exception was the direct cause of the following exception: >>>>> >>>>> Traceback (most recent call last): >>>>> File "/usr/local/lib/pulp/bin/pulpcore-manager", line 8, in <module> >>>>> sys.exit(manage()) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py", >>>>> line 11, in manage >>>>> execute_from_command_line(sys.argv) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py", >>>>> line 381, in execute_from_command_line >>>>> utility.execute() >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py", >>>>> line 375, in execute >>>>> self.fetch_command(subcommand).run_from_argv(self.argv) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/base.py", >>>>> line 323, in run_from_argv >>>>> self.execute(*args, **cmd_options) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/base.py", >>>>> line 364, in execute >>>>> output = self.handle(*args, **options) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/management/commands/handle-artifact-checksums.py", >>>>> line 212, in handle >>>>> artifacts_qs.update(**update_params) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py", >>>>> line 741, in update >>>>> rows = query.get_compiler(self.db).execute_sql(CURSOR) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", >>>>> line 1471, in execute_sql >>>>> cursor = super().execute_sql(result_type) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", >>>>> line 1142, in execute_sql >>>>> cursor.execute(sql, params) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", >>>>> line 67, in execute >>>>> return self._execute_with_wrappers(sql, params, many=False, >>>>> executor=self._execute) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", >>>>> line 76, in _execute_with_wrappers >>>>> return executor(sql, params, many, context) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", >>>>> line 84, in _execute >>>>> return self.cursor.execute(sql, params) >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/utils.py", >>>>> line 89, in __exit__ >>>>> raise dj_exc_value.with_traceback(traceback) from exc_value >>>>> File >>>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", >>>>> line 84, in _execute >>>>> return self.cursor.execute(sql, params) >>>>> django.db.utils.IntegrityError: null value in column "sha1" violates >>>>> not-null constraint >>>>> DETAIL: Failing row contains (1883479d-9e2d-45e1-971f-37246fc3c358, >>>>> 2020-08-13 05:50:33.543549+00, 2020-08-13 05:50:33.543572+00, >>>>> artifact/52/e229d511dca3c15b4250159cc7fd1685e8202cbb20e228bbd6ac..., >>>>> 286400, 51b3d1a3ff20a0f88ccb084eca86073c, null, >>>>> bd036226130d705e95bf4ad9acaa5d8d6d4df7460943062c08154bf1, >>>>> 52e229d511dca3c15b4250159cc7fd1685e8202cbb20e228bbd6ac0f16a27789, >>>>> 53964efdef715dd8f8ff96588e7f1ab184a3747d39bb3c9c28fc339659014274..., >>>>> 0b0d8a7d5d5816705016a5a59489f032fddc31afc438a482549091b730ca4f6b...). >>>>> >>>>> I could just delete the whole bunch and reinstall but that wouldn't >>>>> teach me anything useful I guess so I'm sticking to my guns hoping you >>>>> guys >>>>> can help me out in understanding the root cause. >>>>> >>>>> Kind regards, >>>>> >>>>> *Eric Van Steenbergen* >>>>> >>>>> *E-mail: vs.e...@gmail.com <vs.e...@gmail.com>* >>>>> *Skype: ericvs2014* >>>>> *LinkedIn: Eric Van Steenbergen >>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>>>> >>>>> >>>>> On Wed, Apr 14, 2021 at 3:53 AM Matthias Dellweg <mdell...@redhat.com> >>>>> wrote: >>>>> >>>>>> In your playbook, where there is the list of plugins to install, did >>>>>> you ever remove one from that list? >>>>>> Also can you try adding "upgrade: true" to all plugins in that list? >>>>>> >>>>>> On Wed, Apr 14, 2021 at 5:27 AM Eric VS <vs.e...@gmail.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Me again :-). >>>>>>> >>>>>>> I had a Pulp installation from several weeks ago (on our staging >>>>>>> environment) and wanted to align it today with my main Pulp installation >>>>>>> running 3.12.0. >>>>>>> The staging one was still running on 3.5.0, installed using the >>>>>>> Ansible Pulp Installer so I followed the upgrade process for the >>>>>>> collection >>>>>>> and ran the playbook. This is the error I got, due to version mismatch >>>>>>> but >>>>>>> not sure on how to fix this one. >>>>>>> >>>>>>> RUNNING HANDLER [pulp.pulp_installer.pulp_common : Restore SELinux >>>>>>> contexts on Pulp dirs that may exist] >>>>>>> ************************************************************************************************************************ >>>>>>> changed: [localhost] >>>>>>> >>>>>>> RUNNING HANDLER [pulp.pulp_installer.pulp_common : Collect static >>>>>>> content] >>>>>>> ****************************************************************************************************************************************************** >>>>>>> fatal: [localhost]: FAILED! => {"changed": true, "cmd": >>>>>>> ["/usr/local/lib/pulp/bin/pulpcore-manager", "collectstatic", >>>>>>> "--noinput", >>>>>>> "--link"], "delta": "0:00:00.297250", "end": "2021-04-14 >>>>>>> 11:20:11.886134", >>>>>>> "msg": "non-zero return code", "rc": 1, "start": "2021-04-14 >>>>>>> 11:20:11.588884", "stderr": "Traceback (most recent call last):\n File >>>>>>> \"/usr/local/lib/pulp/bin/pulpcore-manager\", line 8, in <module>\n >>>>>>> sys.exit(manage())\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py\", >>>>>>> line 11, in manage\n execute_from_command_line(sys.argv)\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>> line 381, in execute_from_command_line\n utility.execute()\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>> line 325, in execute\n settings.INSTALLED_APPS\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>> line 79, in __getattr__\n self._setup(name)\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>> line 66, in _setup\n self._wrapped = Settings(settings_module)\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>> line 157, in __init__\n mod = >>>>>>> importlib.import_module(self.SETTINGS_MODULE)\n File >>>>>>> \"/usr/lib64/python3.6/importlib/__init__.py\", line 126, in >>>>>>> import_module\n return _bootstrap._gcd_import(name[level:], package, >>>>>>> level)\n File \"<frozen importlib._bootstrap>\", line 994, in >>>>>>> _gcd_import\n File \"<frozen importlib._bootstrap>\", line 971, in >>>>>>> _find_and_load\n File \"<frozen importlib._bootstrap>\", line 955, in >>>>>>> _find_and_load_unlocked\n File \"<frozen importlib._bootstrap>\", line >>>>>>> 665, in _load_unlocked\n File \"<frozen >>>>>>> importlib._bootstrap_external>\", >>>>>>> line 678, in exec_module\n File \"<frozen importlib._bootstrap>\", line >>>>>>> 219, in _call_with_frames_removed\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>>>> line 83, in <module>\n plugin_app_config = entry_point.load()\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>> line 2317, in load\n self.require(*args, **kwargs)\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>> line 2340, in require\n items = working_set.resolve(reqs, env, >>>>>>> installer, extras=self.extras)\n File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>> line 779, in resolve\n raise VersionConflict(dist, >>>>>>> req).with_context(dependent_req)\npkg_resources.VersionConflict: >>>>>>> (pulpcore >>>>>>> 3.12.0 (/usr/local/lib/pulp/lib/python3.6/site-packages), >>>>>>> Requirement.parse('pulpcore<3.6,>=3.4'))", "stderr_lines": ["Traceback >>>>>>> (most recent call last):", " File >>>>>>> \"/usr/local/lib/pulp/bin/pulpcore-manager\", line 8, in <module>", " >>>>>>> sys.exit(manage())", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/manage.py\", >>>>>>> line 11, in manage", " execute_from_command_line(sys.argv)", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>> line 381, in execute_from_command_line", " utility.execute()", " >>>>>>> File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/management/__init__.py\", >>>>>>> line 325, in execute", " settings.INSTALLED_APPS", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>> line 79, in __getattr__", " self._setup(name)", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>> line 66, in _setup", " self._wrapped = Settings(settings_module)", " >>>>>>> File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/conf/__init__.py\", >>>>>>> line 157, in __init__", " mod = >>>>>>> importlib.import_module(self.SETTINGS_MODULE)", "File >>>>>>> \"/usr/lib64/python3.6/importlib/__init__.py\", line 126, in >>>>>>> import_module", " return _bootstrap._gcd_import(name[level:], >>>>>>> package, >>>>>>> level)", " File \"<frozen importlib._bootstrap>\", line 994, in >>>>>>> _gcd_import", " File \"<frozen importlib._bootstrap>\", line 971, in >>>>>>> _find_and_load", " File \"<frozen importlib._bootstrap>\", line 955, in >>>>>>> _find_and_load_unlocked", " File \"<frozen importlib._bootstrap>\", >>>>>>> line >>>>>>> 665, in _load_unlocked", "File \"<frozen >>>>>>> importlib._bootstrap_external>\", >>>>>>> line 678, in exec_module", " File \"<frozen importlib._bootstrap>\", >>>>>>> line >>>>>>> 219, in _call_with_frames_removed", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/app/settings.py\", >>>>>>> line 83, in <module>", " plugin_app_config = entry_point.load()", " >>>>>>> File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>> line 2317, in load", " self.require(*args, **kwargs)", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>> line 2340, in require", " items = working_set.resolve(reqs, env, >>>>>>> installer, extras=self.extras)", " File >>>>>>> \"/usr/local/lib/pulp/lib64/python3.6/site-packages/pkg_resources/__init__.py\", >>>>>>> line 779, in resolve", " raise VersionConflict(dist, >>>>>>> req).with_context(dependent_req)", "pkg_resources.VersionConflict: >>>>>>> (pulpcore 3.12.0 (/usr/local/lib/pulp/lib/python3.6/site-packages), >>>>>>> Requirement.parse('pulpcore<3.6,>=3.4'))"], "stdout": "", >>>>>>> "stdout_lines": >>>>>>> []} >>>>>>> >>>>>>> RUNNING HANDLER [pulp.pulp_installer.pulp_common : Restart all Pulp >>>>>>> services] >>>>>>> *************************************************************************************************************************************************** >>>>>>> >>>>>>> RUNNING HANDLER [geerlingguy.postgresql : restart postgresql] >>>>>>> ******************************************************************************************************************************************************************* >>>>>>> >>>>>>> NO MORE HOSTS LEFT >>>>>>> ************************************************************************************************************************************************************************************************************** >>>>>>> >>>>>>> PLAY RECAP >>>>>>> ********************************************************************************************************************************************************************************************************************** >>>>>>> localhost : ok=91 changed=33 unreachable=0 >>>>>>> failed=1 skipped=46 rescued=0 ignored=0 >>>>>>> >>>>>>> Any help is greatly appreciated. >>>>>>> >>>>>>> Kind regards, >>>>>>> >>>>>>> *Eric Van Steenbergen* >>>>>>> >>>>>>> *E-mail: vs.e...@gmail.com <vs.e...@gmail.com>* >>>>>>> *Skype: ericvs2014* >>>>>>> *LinkedIn: Eric Van Steenbergen >>>>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>>>>>> _______________________________________________ >>>>>>> Pulp-list mailing list >>>>>>> Pulp-list@redhat.com >>>>>>> https://listman.redhat.com/mailman/listinfo/pulp-list >>>>>> >>>>>> _______________________________________________ > Pulp-list mailing list > Pulp-list@redhat.com > https://listman.redhat.com/mailman/listinfo/pulp-list
_______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-list