Hi Jonathan, I just tried sending SHGCHLD to the parent processes, it didn't have any effect. The parents are "/usr/bin/perl /usr/share/koha/bin/ background_jobs_worker.pl --queue default" and "/usr/bin/perl /usr/share/koha/bin/background_jobs_worker.pl --queue long_tasks".
worker-error.log has a few entries like these three from today: 20230419 08:44:53 ccfls-koha-worker-long_tasks: client (pid 12169) killed by signal 13, respawning 20230419 09:36:06 ccfls-koha-worker: client (pid 14398) killed by signal 13, respawning 20230419 09:59:35 ccfls-koha-worker: client (pid 29935) killed by signal 13, respawning Those timestamps correspond to three jobs in the jobs queue that didn't complete and have a "null/n" (n being numbers that I think correspond to the number of things in the batch). The first is a batch item record modification and the other two are holds queue updates. I cancelled these three jobs and the zombies remained. worker-output.log has a number of entries like these, but unfortunately there are no timestamps so I can't link it to anything, although the timestamp on the file itself is from yesterday at 13:08, which I think corresponded to a successful staging and import of records. Use of uninitialized value $subfield_value in pattern match (m//) at /usr/share/koha/lib/Koha/SimpleMARC.pm line 435. Use of uninitialized value $subfield_value in string eq at /usr/share/koha/lib/Koha/SimpleMARC.pm line 435. I did try something else. The parent process for the long queue had apparently already respawned, but the one for the default one hadn't, so I killed it with -9. The two zombies that had been there went away and the default queue restarted. Before I did that I tried a MARC upload, it was stuck at 0%. I cancelled the job and retried it after killing the default queue and it worked, but it spawned a new zombie which was a child of the long_tasks queue. Yesterday it seemed to work if there was only one zombie, but not two. No new entries in either of the worker- files. Thanks for your help. c. ----------------------------------------------------------- Cindy Murdock Ames IT Services Director Meadville Public Library | CCFLS https://meadvillelibrary.org | https://ccfls.org Please report tech support issues in Mantis: https://mantis.ccfls.org On Wed, Apr 19, 2023 at 2:44 AM Jonathan Druart < jonathan.dru...@bugs.koha-community.org> wrote: > Did you have a look at worker-*.log? Nothing useful there? > > You can try to send SIGCHLD to the parent to kill the zombie. > > Le mar. 18 avr. 2023 à 22:09, Cindy Murdock Ames <cmurd...@ccfls.org> a > écrit : > > > > A few other things I've noticed: > > > > - Sometimes the zombie processes will go away on their own, sometimes it > seems when you retry the MARC import or whatever it was that failed. This > one is really weird to me as in all my years as a sysadmin I thought it was > not possible for zombie processes to go away without a reboot. But maybe > that's changed and now zombies can rise from the dead. Lol. > > > > - In looking at the jobs list in Koha, it seems that Holds queue updates > are especially prone to getting stuck at a progress of null/1. > > > > - If you reattempt a job that is stuck (ie, reattempting a MARC file > upload or what not) it will often succeed. The original failed job remains > with a progress of null. > > > > c. > > ----------------------------------------------------------- > > Cindy Murdock Ames > > IT Services Director > > Meadville Public Library | CCFLS > > https://meadvillelibrary.org | https://ccfls.org > > > > Please report tech support issues in Mantis: https://mantis.ccfls.org > > > > > > On Tue, Apr 18, 2023 at 3:55 PM Cindy Murdock Ames <cmurd...@ccfls.org> > wrote: > >> > >> Yes, it's 22.11.04, package version. > >> > >> ----------------------------------------------------------- > >> Cindy Murdock Ames > >> IT Services Director > >> Meadville Public Library | CCFLS > >> https://meadvillelibrary.org | https://ccfls.org > >> > >> > >> > >> > >> On Tue, Apr 18, 2023 at 2:59 PM Jonathan Druart < > jonathan.dru...@bugs.koha-community.org> wrote: > >>> > >>> Hi Cindy, > >>> Which exact version of Koha 22.11.xx? It should be the latest one. > >>> Regards, > >>> Jonathan > >>> > >>> Le mar. 18 avr. 2023 à 19:13, Cindy Murdock Ames <cmurd...@ccfls.org> > a écrit : > >>> > > >>> > Hi all, > >>> > > >>> > A couple weekends ago I upgraded our Koha instance from 22.05 to > 22.11, and > >>> > I'm having trouble with the background_jobs processes becoming > zombies > >>> > after a very short amount of time, necessitating a reboot. I > suspect it's > >>> > a misconfiguration on my part, so if someone can shed some light I'd > really > >>> > appreciate it! > >>> > > >>> > The first symptom was our MARC imports getting stuck at "import > queued", > >>> > and after some digging (and thanks to the thread in this list with > the > >>> > subject of "Background job / Staging MARC import stuck at 0%" I > found I was > >>> > entirely missing the <message_broker> section in our config, so I > added > >>> > this: > >>> > > >>> > <message_broker> > >>> > <hostname>localhost</hostname> > >>> > <port>61613</port> > >>> > <username>guest</username> > >>> > <password>guest</password> > >>> > <vhost></vhost> > >>> > </message_broker> > >>> > > >>> > Which seemed to resolve it, but now I find that the background_jobs > >>> > processes are going zombie after processing only a few jobs. Here's > some > >>> > info from the rabbitmq log after restarting the server: > >>> > > >>> > =INFO REPORT==== 18-Apr-2023::12:23:46 === > >>> > node : rabbit@ccflskoha > >>> > home dir : /var/lib/rabbitmq > >>> > config file(s) : /etc/rabbitmq/rabbitmq.config (not found) > >>> > cookie hash : ojvkUE6eUtku7kHlx3uiFg== > >>> > log : /var/log/rabbitmq/rab...@ccflskoha.log > >>> > sasl log : /var/log/rabbitmq/rab...@ccflskoha-sasl.log > >>> > database dir : /var/lib/rabbitmq/mnesia/rabbit@ccflskoha > >>> > > >>> > Is it problematic that /etc/rabbitmq/rabbitmq.config is missing? > Anything > >>> > else I should be looking at? We're running on Ubuntu SE 18.04 if > that is > >>> > helpful. > >>> > > >>> > Thanks much! > >>> > Cindy > >>> > > >>> > > >>> > ----------------------------------------------------------- > >>> > Cindy Murdock Ames > >>> > IT Services Director > >>> > Meadville Public Library | CCFLS > >>> > https://meadvillelibrary.org | https://ccfls.org > >>> > _______________________________________________ > >>> > > >>> > Koha mailing list http://koha-community.org > >>> > Koha@lists.katipo.co.nz > >>> > Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha > _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
