On Tue, Apr 13, 2021 at 4:47 AM Eric VS <[email protected]> wrote: > Hi Gerrod, > > Update.... I found how to do the upgrade... ansible-galaxy collections > install ... --force-with-deps. Now I'm running 3.12 but get the following > error. > > ╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) pulp status > > > 127 ↵ > Notice: Cached api is outdated. Refreshing... > { > "versions": [ > { > "component": "core", > "version": "3.12.0" > }, > { > "component": "rpm", > "version": "3.10.0" > } > ], > "online_workers": [ > { > "pulp_href": > "/pulp/api/v3/workers/179af4ab-6846-431c-bc51-a07db3ea9a1b/", > "pulp_created": "2021-04-13T02:41:05.521531Z", > "name": "[email protected]", > "last_heartbeat": "2021-04-13T02:41:50.701103Z" > }, > { > "pulp_href": > "/pulp/api/v3/workers/77c08991-a27a-49d6-a2c7-87cd8c26877c/", > "pulp_created": "2021-03-31T07:59:28.269742Z", > "name": "resource-manager", > "last_heartbeat": "2021-04-13T02:41:51.803953Z" > }, > { > "pulp_href": > "/pulp/api/v3/workers/75bc2422-2f58-41cb-93d2-17b69732d946/", > "pulp_created": "2021-04-13T02:41:08.214845Z", > "name": "[email protected]", > "last_heartbeat": "2021-04-13T02:41:53.709207Z" > } > ], > "online_content_apps": [ > { > "name": "[email protected]", > "last_heartbeat": "2021-04-13T02:41:56.489400Z" > }, > { > "name": "[email protected]", > "last_heartbeat": "2021-04-13T02:41:56.646680Z" > }, > { > "name": "[email protected]", > "last_heartbeat": "2021-04-13T02:41:56.918615Z" > }, > { > "name": "[email protected]", > "last_heartbeat": "2021-04-13T02:41:57.224724Z" > }, > { > "name": "[email protected]", > "last_heartbeat": "2021-04-13T02:41:57.319709Z" > }, > { > "name": "[email protected]", > "last_heartbeat": "2021-04-13T02:41:57.328655Z" > }, > { > "name": "[email protected]", > "last_heartbeat": "2021-04-13T02:41:55.609724Z" > }, > { > "name": "[email protected]", > "last_heartbeat": "2021-04-13T02:41:55.824550Z" > } > ], > "database_connection": { > "connected": true > }, > "redis_connection": { > "connected": true > }, > "storage": { > "total": 161008844800, > "used": 109266341888, > "free": 51742502912 > } > } > > (ansible-venv) ╭─ericv@ip-10-80-1-57 ~ > ╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) http > https://pulp-dev.activeops.io/pulp/content/ > > http: error: TypeError: __str__ returned non-string (type Error) > It keeps getting better. This looks to me like an error in httpie (the package that provides http). If you are running that from a virtualenv, can you make sure it is properly installed with `pip install --upgrade httpie`? Otherwise, can you retry with curl?
Don't worry for the timezone, we just _know_ that problem, or is it a convenience... > > Kind regards, > > *Eric Van Steenbergen* > > *E-mail: [email protected] <[email protected]>* > *Skype: ericvs2014* > *LinkedIn: Eric Van Steenbergen > <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* > > > On Mon, Apr 12, 2021 at 8:35 PM Eric VS <[email protected]> wrote: > >> Hello Gerrod, >> >> Thank you for the update. On the node that errors I have indeed 3.11 >> installed, on another one I'm doing right now I have 3.12. >> Dumb question, I'm not that familiar with ansible-galaxy. How do I >> 'upgrade' pulp_installer? I tried to run the install command again which >> just says pulp.installer is already installed. Even downloading and trying >> to install from the file gives the same result. >> >> Kind regards, >> >> *Eric Van Steenbergen* >> >> *E-mail: [email protected] <[email protected]>* >> *Skype: ericvs2014* >> *LinkedIn: Eric Van Steenbergen >> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >> >> >> On Mon, Apr 12, 2021 at 8:28 PM Gerrod Ubben <[email protected]> wrote: >> >>> Hi Eric, >>> >>> This bug [0] was fixed in pulpcore 3.12 which we just finished releasing >>> today. You can use the 3.12 pulp_installer [1] to upgrade to pulpcore 3.12 >>> or change the erroring line [2] in your current installation if you don't >>> want to upgrade right away. Also, all of Pulp's API endpoints require a >>> trailing forward slash or else you'll get a 301 return. So to hit the >>> status endpoint you use '/pulp/api/v3/status/', but we encourage everyone >>> to use the CLI and make feature requests [3] if you find missing >>> functionality. >>> >>> Gerrod >>> >>> [0] https://pulp.plan.io/issues/8475 >>> [1] https://galaxy.ansible.com/pulp/pulp_installer >>> [2] https://git.io/JOqh0 >>> [3] https://github.com/pulp/pulp-cli/issues/new/ >>> >>> On Mon, Apr 12, 2021 at 8:46 PM Eric VS <[email protected]> wrote: >>> >>>> Hello Matthias and David, >>>> >>>> Thanks again for your swift responses, I'm in a different time zone >>>> most likely so apologies for late replies. >>>> >>>> @Matthias Dellweg <[email protected]> , when running the v3/status I >>>> get this: >>>> >>>> ╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) http >>>> https://pulp-dev.activeops.io/pulp/api/v3/status >>>> HTTP/1.1 301 Moved Permanently >>>> Access-Control-Expose-Headers: Correlation-ID >>>> Connection: keep-alive >>>> Content-Length: 0 >>>> Content-Type: text/html; charset=utf-8 >>>> Correlation-ID: 4a98ac4213d44645ae9c963c81873780 >>>> Date: Tue, 13 Apr 2021 00:40:05 GMT >>>> Location: /pulp/api/v3/status/ >>>> Server: nginx/1.16.1 >>>> Strict-Transport-Security: max-age=15768000 >>>> (ansible-venv) ╭─ericv@ip-10-80-1-57 ~ >>>> >>>> Tracing journalctl gives me the below when I run the /pulp/content with >>>> http. >>>> >>>> ╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) sudo journalctl -f >>>> -u pulpcore-content >>>> -- Logs begin at Thu 2020-06-04 07:04:49 UTC. -- >>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: File >>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_protocol.py", >>>> line 422, in _handle_request >>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: resp = await self._request_handler(request) >>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: File >>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_app.py", >>>> line 499, in _handle >>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: resp = await handler(request) >>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: File >>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py", >>>> line 121, in list_distributions >>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in >>>> base_paths] >>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: File >>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py", >>>> line 121, in <listcomp> >>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in >>>> base_paths] >>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: AttributeError: 'str' object has no attribute 'base_path' >>>> Apr 13 00:42:45 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: 127.0.0.1 [13/Apr/2021:00:42:45 +0000] "GET /pulp/content/ >>>> HTTP/1.0" 500 231 "-" "HTTPie/0.9.4" >>>> >>>> >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: [2021-04-13 00:43:40 +0000] [1452] [ERROR] Error handling >>>> request >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: Traceback (most recent call last): >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: File >>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_protocol.py", >>>> line 422, in _handle_request >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: resp = await self._request_handler(request) >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: File >>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/web_app.py", >>>> line 499, in _handle >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: resp = await handler(request) >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: File >>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py", >>>> line 121, in list_distributions >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in >>>> base_paths] >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: File >>>> "/usr/local/lib/pulp/lib64/python3.6/site-packages/pulpcore/content/handler.py", >>>> line 121, in <listcomp> >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: directory_list = ["{}/".format(d.base_path) for d in >>>> base_paths] >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: AttributeError: 'str' object has no attribute 'base_path' >>>> Apr 13 00:43:40 ip-10-80-1-57.ap-northeast-1.compute.internal >>>> gunicorn[1216]: 127.0.0.1 [13/Apr/2021:00:43:40 +0000] "GET /pulp/content/ >>>> HTTP/1.0" 500 231 "-" "HTTPie/0.9.4" >>>> >>>> Kind regards, >>>> >>>> *Eric Van Steenbergen* >>>> >>>> *E-mail: [email protected] <[email protected]>* >>>> *Skype: ericvs2014* >>>> *LinkedIn: Eric Van Steenbergen >>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>>> >>>> >>>> On Mon, Apr 12, 2021 at 9:55 AM David Davis <[email protected]> >>>> wrote: >>>> >>>>> It could potentially be this bug which was fixed in pulpcore 3.12: >>>>> https://pulp.plan.io/issues/8447 >>>>> >>>>> The stacktrace would be helpful though to know for sure. >>>>> >>>>> David >>>>> >>>>> >>>>> On Mon, Apr 12, 2021 at 5:27 AM Matthias Dellweg <[email protected]> >>>>> wrote: >>>>> >>>>>> Can you monitor "journalctl -f -u pulpcore-content" while performing >>>>>> that call? >>>>>> >>>>>> On Mon, Apr 12, 2021 at 11:17 AM Matthias Dellweg < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Sorry, typo (e and 3 are too close on my k3yboard). Should have been >>>>>>> .../pulp/api/v3/status >>>>>>> >>>>>>> On Mon, Apr 12, 2021 at 11:06 AM Eric VS <[email protected]> wrote: >>>>>>> >>>>>>>> Hey Matthias, >>>>>>>> >>>>>>>> I installed Pulp using the pulp_installer Ansible playbook, no >>>>>>>> errors. >>>>>>>> >>>>>>>> This is the output from `pulp status` since the query using http >>>>>>>> didn't work either: >>>>>>>> >>>>>>>> ╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) pulp status >>>>>>>> { >>>>>>>> "versions": [ >>>>>>>> { >>>>>>>> "component": "core", >>>>>>>> "version": "3.11.0" >>>>>>>> }, >>>>>>>> { >>>>>>>> "component": "rpm", >>>>>>>> "version": "3.10.0" >>>>>>>> } >>>>>>>> ], >>>>>>>> "online_workers": [ >>>>>>>> { >>>>>>>> "pulp_created": "2021-03-31T07:59:28.269742Z", >>>>>>>> "pulp_href": >>>>>>>> "/pulp/api/v3/workers/77c08991-a27a-49d6-a2c7-87cd8c26877c/", >>>>>>>> "name": "resource-manager", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:03.424149Z" >>>>>>>> }, >>>>>>>> { >>>>>>>> "pulp_created": "2021-04-05T06:46:11.689705Z", >>>>>>>> "pulp_href": >>>>>>>> "/pulp/api/v3/workers/ed4c9dca-db6d-4d52-9f0b-331a937d2755/", >>>>>>>> "name": "[email protected]", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:03.624644Z" >>>>>>>> }, >>>>>>>> { >>>>>>>> "pulp_created": "2021-04-05T06:46:11.689909Z", >>>>>>>> "pulp_href": >>>>>>>> "/pulp/api/v3/workers/9d538294-a3aa-48d4-95ab-e0a213bae94f/", >>>>>>>> "name": "[email protected]", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:11.745839Z" >>>>>>>> } >>>>>>>> ], >>>>>>>> "online_content_apps": [ >>>>>>>> { >>>>>>>> "name": "[email protected]", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:10.886966Z" >>>>>>>> }, >>>>>>>> { >>>>>>>> "name": "[email protected]", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:13.566778Z" >>>>>>>> }, >>>>>>>> { >>>>>>>> "name": "[email protected]", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:13.568876Z" >>>>>>>> }, >>>>>>>> { >>>>>>>> "name": "[email protected]", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:15.876728Z" >>>>>>>> }, >>>>>>>> { >>>>>>>> "name": "[email protected]", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:15.881384Z" >>>>>>>> }, >>>>>>>> { >>>>>>>> "name": "[email protected]", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:15.881615Z" >>>>>>>> }, >>>>>>>> { >>>>>>>> "name": "[email protected]", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:15.883369Z" >>>>>>>> }, >>>>>>>> { >>>>>>>> "name": "[email protected]", >>>>>>>> "last_heartbeat": "2021-04-12T09:02:15.885268Z" >>>>>>>> } >>>>>>>> ], >>>>>>>> "database_connection": { >>>>>>>> "connected": true >>>>>>>> }, >>>>>>>> "redis_connection": { >>>>>>>> "connected": true >>>>>>>> }, >>>>>>>> "storage": { >>>>>>>> "total": 161008844800, >>>>>>>> "used": 109256830976, >>>>>>>> "free": 51752013824 >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> For reference this is the query you requested using api/ve/status >>>>>>>> >>>>>>>> ╰─$ (⎈ k8s01.aops-dev.activeops.local:kube-system) http >>>>>>>> https://pulp-dev.activeops.io/pulp/api/ve/status >>>>>>>> HTTP/1.1 404 Not Found >>>>>>>> Access-Control-Expose-Headers: Correlation-ID >>>>>>>> Connection: keep-alive >>>>>>>> Content-Length: 77 >>>>>>>> Content-Type: text/html >>>>>>>> Correlation-ID: 14cab8b25dc64591a243fd391424924f >>>>>>>> Date: Mon, 12 Apr 2021 09:04:06 GMT >>>>>>>> Server: nginx/1.16.1 >>>>>>>> X-Frame-Options: SAMEORIGIN >>>>>>>> >>>>>>>> <h1>Not Found</h1><p>The requested resource was not found on this >>>>>>>> server.</p> >>>>>>>> >>>>>>>> Kind regards, >>>>>>>> >>>>>>>> *Eric Van Steenbergen* >>>>>>>> >>>>>>>> *E-mail: [email protected] <[email protected]>* >>>>>>>> *Skype: ericvs2014* >>>>>>>> *LinkedIn: Eric Van Steenbergen >>>>>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Apr 12, 2021 at 4:00 AM Matthias Dellweg < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Eric, >>>>>>>>> can you provide more info on this? >>>>>>>>> Helpful would be: >>>>>>>>> How is the server installed? (container, pulp_installer, by hand >>>>>>>>> from pypi) >>>>>>>>> What versions of the components are installed? (http https:// >>>>>>>>> <FQDN>/pulp/api/ve/status) >>>>>>>>> Thanks in advance! >>>>>>>>> >>>>>>>>> On Mon, Apr 12, 2021 at 10:57 AM Eric VS <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Matthias, >>>>>>>>>> >>>>>>>>>> When I run `http https://<FQDN>/pulp/content/` is when I get the >>>>>>>>>> 500 error. >>>>>>>>>> I can query the all the content with the API (pulp rpm >>>>>>>>>> repository/remote/publication/distribution) without any issue. >>>>>>>>>> >>>>>>>>>> Kind regards, >>>>>>>>>> >>>>>>>>>> *Eric Van Steenbergen* >>>>>>>>>> >>>>>>>>>> *E-mail: [email protected] <[email protected]>* >>>>>>>>>> *Skype: ericvs2014* >>>>>>>>>> *LinkedIn: Eric Van Steenbergen >>>>>>>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Apr 12, 2021 at 3:48 AM Matthias Dellweg < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Can you isolate the command that leads to this server response? >>>>>>>>>>> >>>>>>>>>>> On Mon, Apr 12, 2021 at 1:55 AM Eric VS <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I've been setting up another Pulp server, replicating >>>>>>>>>>>> repositories from my main one for 'local' use in an environment. >>>>>>>>>>>> When I >>>>>>>>>>>> was adding remotes and repositories at one point in time I started >>>>>>>>>>>> getting >>>>>>>>>>>> the below error: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *HTTP/1.1 500 Internal Server ErrorConnection: >>>>>>>>>>>> keep-aliveContent-Length: 55Content-Type: text/plain; >>>>>>>>>>>> charset=utf-8Date: >>>>>>>>>>>> Sun, 11 Apr 2021 23:53:14 GMTServer: nginx/1.16.1500 Internal >>>>>>>>>>>> Server >>>>>>>>>>>> ErrorServer got itself in trouble* >>>>>>>>>>>> >>>>>>>>>>>> I have enabled debug on Nginx but can't see anything indicating >>>>>>>>>>>> on why this all of a sudden fails. >>>>>>>>>>>> >>>>>>>>>>>> Any help is greatly appreciated. >>>>>>>>>>>> >>>>>>>>>>>> Kind regards, >>>>>>>>>>>> >>>>>>>>>>>> *Eric Van Steenbergen* >>>>>>>>>>>> >>>>>>>>>>>> *E-mail: [email protected] <[email protected]>* >>>>>>>>>>>> *Skype: ericvs2014* >>>>>>>>>>>> *LinkedIn: Eric Van Steenbergen >>>>>>>>>>>> <http://es.linkedin.com/pub/eric-van-steenbergen/b/8a4/51b>* >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Pulp-list mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> https://listman.redhat.com/mailman/listinfo/pulp-list >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>> Pulp-list mailing list >>>>>> [email protected] >>>>>> https://listman.redhat.com/mailman/listinfo/pulp-list >>>>> >>>>> _______________________________________________ >>>> Pulp-list mailing list >>>> [email protected] >>>> https://listman.redhat.com/mailman/listinfo/pulp-list >>> >>> _______________________________________________ > Pulp-list mailing list > [email protected] > https://listman.redhat.com/mailman/listinfo/pulp-list
_______________________________________________ Pulp-list mailing list [email protected] https://listman.redhat.com/mailman/listinfo/pulp-list
