Neil,
Pulp is really backed up with tasks, my guess would be there was some sort
of deadlock in qpid and it "unlocked", releasing all of these tasks. My
advice would be first try stopping then starting these services
pulp_celerybeat.service pulp_workers.service pulp_resource_manager.service,
and then restarting qpidd and httpd. Wait a bit and see if tasks start
clearing out.
If that doesn't work and you want to go a more forceful route, you can try
deleting the tasks in pulp. This can cause some serious issues if you have
open tasks that you actually care about (like a repo sync), so if you do
wind up doing this, make sure any tasks you care about are finished up.
You've been warned :)
pulp-admin -u admin -p $pulpAdminPassword tasks list | grep "Task Id"| awk
'{print $3}' | xargs -I % pulp-admin -u admin -p $pulpAdminPassword tasks
cancel --task-id %
If you get unstuck, I would also advise upgrading to 3.4. Let me know how
it goes!
-John
John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch
On Thu, Jun 15, 2017 at 10:13 AM, Neil Hanlon <nhan...@kayak.com> wrote:
> Thanks for taking a look for me. Here's the output of the tasks list.
>
>
> On Thu, Jun 15, 2017 at 10:07 AM John Mitsch <jomit...@redhat.com> wrote:
>
>> Seems like something funny is going on with the pulp tasks, can you
>> install pulp-admin and take a look at the tasks? We may need to kill
>> something at that level
>>
>> to install: https://gist.github.com/johnpmitsch/
>> 866fb430780ba0e95a5554566b989416
>>
>> -John
>>
>> John Mitsch
>> Red Hat Engineering
>> (860)-967-7285 <(860)%20967-7285>
>> irc: jomitsch
>>
>> On Tue, Jun 13, 2017 at 5:06 PM, Neil Hanlon <nhan...@kayak.com> wrote:
>>
>>> Certainly. Going to let that run overnight :)
>>>
>>> On Tue, Jun 13, 2017 at 5:00 PM John Mitsch <jomit...@redhat.com> wrote:
>>>
>>>> That is very strange, this may be a good candidate for a foreman-debug
>>>> so we can dig through logs. Can you create one and send it to me privately?
>>>>
>>>> -John
>>>>
>>>> John Mitsch
>>>> Red Hat Engineering
>>>> (860)-967-7285 <(860)%20967-7285>
>>>> irc: jomitsch
>>>>
>>>> On Tue, Jun 13, 2017 at 3:34 PM, <nhan...@kayak.com> wrote:
>>>>
>>>>> So mostly we use foreman/katello just for syncing operating systems
>>>>> out to capsules/smart proxies, and for managing system lifecycle
>>>>> (basically, just kickstarts). I went last week to add a XenServer 7.2 OS
>>>>> into my Katello Products, and started having tons of issues, specifically
>>>>> in that I had 20k+ 'Actions::Katello::Host::UploadPackageProfile'
>>>>> tasks sitting in my list as state = running, result = pending--but not
>>>>> doing anything at all.
>>>>>
>>>>> To be honest, I don't really care why they're not working--I just want
>>>>> them to stop being scheduled at all. Today I've shut off foreman-tasks and
>>>>> pulp_workers and tried to set all these tasks which were running/pending
>>>>> to
>>>>> stopped/error, and that worked. Seemed to help out with the issues of
>>>>> Foreman. However I still had some tasks that weren't being scheduled, so I
>>>>> checked and sure enough, foreman tasks was running, as was pulp, but it
>>>>> was
>>>>> complaining about not having a dynflow environment. Restarting foreman
>>>>> tasks and waiting a few minutes helped, at which point I started seeing
>>>>> stuff like this spam my production.log, and tons of
>>>>> Actions::Katello::Host::UploadPackageProfile tasks being scheduled
>>>>> again--with dates from *months* ago.
>>>>>
>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
>>>>> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event
>>>>> Dynflow::Action::Polling::Poll
>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
>>>>> c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
>>>>> df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
>>>>> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
>>>>> f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step
>>>>> dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "
>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/
>>>>> 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5,
>>>>> application/xml", "Accept-Encoding"=>"gzip, deflate",
>>>>> "accept"=>"application/json", "content_type"=>"application/json"
>>>>> | \nRestClient.get "https://c4katello02.c4.<
>>>>> redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/",
>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
>>>>> deflate", "accept"=>"application/json", "content_type"=>"application/
>>>>> json"
>>>>> | \n# => 200 OK | application/json 326 bytes
>>>>> |
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "
>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/
>>>>> 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5,
>>>>> application/xml", "Accept-Encoding"=>"gzip, deflate",
>>>>> "accept"=>"application/json", "content_type"=>"application/json"
>>>>> | \nRestClient.get "https://c4katello02.c4.<
>>>>> redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/",
>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
>>>>> deflate", "accept"=>"application/json", "content_type"=>"application/
>>>>> json"
>>>>> | \n# => 200 OK | application/json 326 bytes
>>>>> | \n# => 200 OK | application/json 328 bytes
>>>>> |
>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "
>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/
>>>>> 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5,
>>>>> application/xml", "Accept-Encoding"=>"gzip, deflate",
>>>>> "accept"=>"application/json", "content_type"=>"application/json"
>>>>> | \nRestClient.get "https://c4katello02.c4.<
>>>>> redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/",
>>>>> "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip,
>>>>> deflate", "accept"=>"application/json", "content_type"=>"application/
>>>>> json"
>>>>> | \n# => 200 OK | application/json 326 bytes
>>>>> | \n# => 200 OK | application/json 328 bytes
>>>>> | \n# => 200 OK | application/json 328 bytes
>>>>> |
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event
>>>>> Dynflow::Action::Polling::Poll
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 2392d32f-0886-4473-ae5a-7595504acb39: 4 got event
>>>>> Dynflow::Action::Polling::Poll
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event
>>>>> Dynflow::Action::Polling::Poll
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event
>>>>> Dynflow::Action::Polling::Poll
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event
>>>>> Dynflow::Action::Polling::Poll
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "
>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/
>>>>> f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5,
>>>>> application/xml", "Accept-Encoding"=>"gzip, deflate",
>>>>> "accept"=>"application/json", "content_type"=>"application/json"
>>>>> | \n# => 200 OK | application/json 329 bytes
>>>>> |
>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "
>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/
>>>>> f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5,
>>>>> application/xml", "Accept-Encoding"=>"gzip, deflate",
>>>>> "accept"=>"application/json", "content_type"=>"application/json"
>>>>> | \n# => 200 OK | application/json 329 bytes
>>>>> | \n# => 200 OK | application/json 329 bytes
>>>>> |
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "
>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/
>>>>> f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5,
>>>>> application/xml", "Accept-Encoding"=>"gzip, deflate",
>>>>> "accept"=>"application/json", "content_type"=>"application/json"
>>>>> | \n# => 200 OK | application/json 329 bytes
>>>>> | \n# => 200 OK | application/json 329 bytes
>>>>> | \n# => 200 OK | application/json 330 bytes
>>>>> |
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event
>>>>> Dynflow::Action::Polling::Poll
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "
>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/
>>>>> f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5,
>>>>> application/xml", "Accept-Encoding"=>"gzip, deflate",
>>>>> "accept"=>"application/json", "content_type"=>"application/json"
>>>>> | \n# => 200 OK | application/json 329 bytes
>>>>> | \n# => 200 OK | application/json 329 bytes
>>>>> | \n# => 200 OK | application/json 330 bytes
>>>>> | \n# => 200 OK | application/json 327 bytes
>>>>> |
>>>>> 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step
>>>>> 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in
>>>>> phase Run Actions::Pulp::Consumer::GenerateApplicability
>>>>> 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "
>>>>> https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/
>>>>> f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5,
>>>>> application/xml", "Accept-Encoding"=>"gzip, deflate",
>>>>> "accept"=>"application/json", "content_type"=>"application/json"
>>>>> | \n# => 200 OK | application/json 329 bytes
>>>>> | \n# => 200 OK | application/json 329 bytes
>>>>>
>>>>>
>>>>>
>>>>> <https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png>
>>>>>
>>>>>
>>>>>
>>>>> I'm happy to upload any logs anyone might want to look at to help me
>>>>> with this.. but I'm at my wits end with this. I'm about ready to tear it
>>>>> all down and just rebuild it from scratch...
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Foreman users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to foreman-users+unsubscr...@googlegroups.com.
>>>>> To post to this group, send email to foreman-users@googlegroups.com.
>>>>> Visit this group at https://groups.google.com/group/foreman-users.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Foreman users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to foreman-users+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to foreman-users@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/foreman-users.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Foreman users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to foreman-users+unsubscr...@googlegroups.com.
>>> To post to this group, send email to foreman-users@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/foreman-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Foreman users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to foreman-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/foreman-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Foreman users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-users+unsubscr...@googlegroups.com.
> To post to this group, send email to foreman-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/foreman-users.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to foreman-users+unsubscr...@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
