https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=44024
--- Comment #1 from Johan Larsson <[email protected]> --- Created attachment 210828 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=210828&action=edit Bug 44024: es_indexer_daemon.pl: retry the batch while Elasticsearch is unreachable When Elasticsearch cannot be reached, commit() hands the jobs back to the queue (status 'new') and returns. That cannot work for the background jobs queue in general: - with JobsNotificationMethod = STOMP the broker message has already been acknowledged before commit() runs, and the daemon only reads the background_jobs table in polling mode, so a job reset to 'new' is never picked up again; - the batch is dropped from the worker, so nothing is retried until something else notices the 'new' job. Instead keep the batch in the forked worker that already owns it and retry it every ES_RETRY_DELAY seconds (default 10) for at most ES_RETRY_MAX_WAIT seconds (default 3600). The jobs stay 'started' while waiting, and finish as soon as Elasticsearch answers. After ES_RETRY_MAX_WAIT they are marked 'failed', like any indexing error. A SIGTERM or SIGINT while waiting marks the jobs 'failed' and exits, and so does a worker whose parent daemon has gone (a stop that only signals the parent), instead of waiting on as an orphan. Errors are classified by is_connection_error(): NoNodes, Timeout, Cxn and Unavailable, plus transport failures reported with the pseudo status 599 (HTTP::Tiny), which Search::Elasticsearch reports as type Request when an open connection is reset. Real indexing errors still mark the jobs failed. The indexing of a batch moves to index_batch() so it can be called again. Test plan: 1. Start misc/workers/es_indexer_daemon.pl (polling or STOMP). 2. Stop Elasticsearch. 3. Modify a record so that an indexing job is enqueued. => The job stays 'started'; the worker log shows 'Elasticsearch unavailable, retrying 1 job(s) every 10s'. 4. Start Elasticsearch again. => The log shows 'Elasticsearch reachable again, 1 job(s) indexed after Ns', the job finishes and the record is searchable. 5. Repeat 1-3 and send SIGTERM to the worker. => The job is marked 'failed' and the worker exits. The same happens when only the parent daemon is sent SIGTERM. 6. Repeat 1-3 with ES_RETRY_MAX_WAIT=30. => After 30 seconds the job is marked 'failed'. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. _______________________________________________ Koha-bugs mailing list -- [email protected] To unsubscribe send an email to [email protected] website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
