That is very strange, this may be a good candidate for a foreman-debug so we can dig through logs. Can you create one and send it to me privately?
-John John Mitsch Red Hat Engineering (860)-967-7285 irc: jomitsch On Tue, Jun 13, 2017 at 3:34 PM, <nhan...@kayak.com> wrote: > So mostly we use foreman/katello just for syncing operating systems out to > capsules/smart proxies, and for managing system lifecycle (basically, just > kickstarts). I went last week to add a XenServer 7.2 OS into my Katello > Products, and started having tons of issues, specifically in that I had > 20k+ 'Actions::Katello::Host::UploadPackageProfile' tasks sitting in my > list as state = running, result = pending--but not doing anything at all. > > To be honest, I don't really care why they're not working--I just want > them to stop being scheduled at all. Today I've shut off foreman-tasks and > pulp_workers and tried to set all these tasks which were running/pending to > stopped/error, and that worked. Seemed to help out with the issues of > Foreman. However I still had some tasks that weren't being scheduled, so I > checked and sure enough, foreman tasks was running, as was pulp, but it was > complaining about not having a dynflow environment. Restarting foreman > tasks and waiting a few minutes helped, at which point I started seeing > stuff like this spam my production.log, and tons of > Actions::Katello::Host::UploadPackageProfile tasks being scheduled > again--with dates from *months* ago. > > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > df8fcf4c-0de7-4cca-8aaf-f38adcd2b6ec: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > f96f15cb-5933-4b18-956d-a2e4c7052004: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:21 [foreman-tasks/dynflow] [D] Step > dc8976be-99e7-4dc4-8a3f-893d3687a1cc: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \nRestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 326 bytes > | > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > c0e798bf-a9f9-40c7-9d55-75d686454f42: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \nRestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 326 bytes > | \n# => 200 OK | application/json 328 bytes > | > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > 41b06756-a517-415b-bcb9-9002dc0e5fec/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \nRestClient.get "https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > ed6df62c-4f20-41c7-a0e1-4fb1cf280e34/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 326 bytes > | \n# => 200 OK | application/json 328 bytes > | \n# => 200 OK | application/json 328 bytes > | > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > f96f15cb-5933-4b18-956d-a2e4c7052004: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > a4967ad4-cd92-4ed3-af38-a88a4c7d35ef: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 2392d32f-0886-4473-ae5a-7595504acb39: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 2392d32f-0886-4473-ae5a-7595504acb39: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > a03805a6-1725-4a92-b3f5-05514cc52c81: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > a03805a6-1725-4a92-b3f5-05514cc52c81: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 96b10629-bfed-4cc5-a9b8-b80e1fc5b302: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 329 bytes > | > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 329 bytes > | > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 1391dcca-6002-434c-a66d-b1dfe4d4628b: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 2392d32f-0886-4473-ae5a-7595504acb39: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 330 bytes > | > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > a03805a6-1725-4a92-b3f5-05514cc52c81: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 got event > Dynflow::Action::Polling::Poll > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 3c827032-6d3b-4a5b-975c-cf78ec6268bd: 4 suspended >> running in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 330 bytes > | \n# => 200 OK | application/json 327 bytes > | > 2017-06-13 03:35:23 [foreman-tasks/dynflow] [D] Step > 903f497f-a539-45af-a220-4cf3d7dd70e2: 4 running >> suspended in phase > Run Actions::Pulp::Consumer::GenerateApplicability > 2017-06-13 03:35:23 [katello/pulp_rest] [D] RestClient.get " > https://c4katello02.c4.<redacted>/pulp/api/v2/tasks/ > f2f90682-f0c9-404d-98d7-8227fceda2b7/", "Accept"=>"*/*; q=0.5, > application/xml", "Accept-Encoding"=>"gzip, deflate", > "accept"=>"application/json", "content_type"=>"application/json" > | \n# => 200 OK | application/json 329 bytes > | \n# => 200 OK | application/json 329 bytes > > > > <https://lh3.googleusercontent.com/-sEewHxc4UuA/WUA9z6RVaWI/AAAAAAAABJM/mWeOvhRvpgknjIOZs8Unwt1l2791tugbACLcBGAs/s1600/Selection_133.png> > > > > I'm happy to upload any logs anyone might want to look at to help me with > this.. but I'm at my wits end with this. I'm about ready to tear it all > down and just rebuild it from scratch... > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-users+unsubscr...@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at https://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Foreman users" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscr...@googlegroups.com. To post to this group, send email to foreman-users@googlegroups.com. Visit this group at https://groups.google.com/group/foreman-users. For more options, visit https://groups.google.com/d/optout.
