So mostly we use foreman/katello just for syncing operating systems out to capsules/smart proxies, and for managing system lifecycle (basically, just kickstarts). I went last week to add a XenServer 7.2 OS into my Katello Products, and started having tons of issues, specifically in that I had 20k+ 'Actions::Katello::Host::UploadPackageProfile' tasks sitting in my list as state = running, result = pending--but not doing anything at all.
To be honest, I don't really care why they're not working--I just want them to stop being scheduled at all. Today I've shut off foreman-tasks and pulp_workers and tried to set all these tasks which were running/pending to stopped/error, and that worked. Seemed to help out with the issues of Foreman. However I still had some tasks that weren't being scheduled, so I checked and sure enough, foreman tasks was running, as was pulp, but it was complaining about not having a dynflow environment. Restarting foreman tasks and waiting a few minutes helped, at which point I started seeing stuff like this spam my production.log, and tons of Actions::Katello::Host::UploadPackageProfile tasks being scheduled again--with dates from *months* ago. 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event Dynflow::Action::Polling::Poll 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \nRestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 200 OK | application/json 326 bytes | 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \nRestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 200 OK | application/json 326 bytes | \n# => 200 OK | application/json 328 bytes | 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \nRestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 200 OK | application/json 326 bytes | \n# => 200 OK | application/json 328 bytes | \n# => 200 OK | application/json 328 bytes | 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event Dynflow::Action::Polling::Poll 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 2392d32f-0886-4473-ae5a-7595504acb39: 4 got event Dynflow::Action::Polling::Poll 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event Dynflow::Action::Polling::Poll 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event Dynflow::Action::Polling::Poll 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event Dynflow::Action::Polling::Poll 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 200 OK | application/json 329 bytes | 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 200 OK | application/json 329 bytes | \n# => 200 OK | application/json 329 bytes | 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 200 OK | application/json 329 bytes | \n# => 200 OK | application/json 329 bytes | \n# => 200 OK | application/json 330 bytes | 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event Dynflow::Action::Polling::Poll 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 200 OK | application/json 329 bytes | \n# => 200 OK | application/json 329 bytes | \n# => 200 OK | application/json 330 bytes | \n# => 200 OK | application/json 327 bytes | 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase Run Actions::Pulp::Consumer::GenerateApplicability 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 200 OK | application/json 329 bytes | \n# => 200 OK | application/json 329 bytes <https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png> I'm happy to upload any logs anyone might want to look at to help me with this.. but I'm at my wits end with this. I'm about ready to tear it all down and just rebuild it from scratch... -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscr...@googlegroups.com. To post to this group, send email to foreman-users@googlegroups.com. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
