https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41307
--- Comment #12 from Matt Blenkinsop <[email protected]> --- Created attachment 209225 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=209225&action=edit Bug 41307: Record the outcome of every SUSHI harvest in the counter logs Until now a counter log row was only written after a COUNTER file was stored, and logdetails was always empty. A harvest that failed left no record, and any exception in the SUSHI response blocked the import, including warnings such as 3040 Partial Data Returned. This patch writes one counter log row for every harvest attempt and for every file upload: - harvest_sushi collects the exceptions of the SUSHI response (top level exception, Exceptions array, Report_Header.Exceptions, vendor error bodies) and classifies the outcome as success, no_usage, no_data, retry or error. Code 3030 No Usage Available without items gives no_usage: the month is complete, the usage is zero. Code 3032, HTTP 204 and an empty report give no_data. Retry codes are 1000, 1010, 1011, 1020, 3031 and 3040, plus HTTP 202, 429 and 5xx. The request parameter warnings 3050, 3060, 3061 and 3062 are informational: the report is complete. A response that is not JSON is an error. - Reports with items are stored even when they carry a warning exception; the log is flagged retry for 3040 and 3031. A report with items and an error exception is not imported: it is logged as error. - logdetails holds a JSON document with the HTTP status, the exceptions (code, severity, message, data), an optional free text message and the COUNTER release. The job messages carry the exception message and data. The request URL is never stored or logged, it holds the API key. The HTTP failure is logged through Koha::Logger instead of warn. - A harvest without a file has a log row without a file name. One row per background job: a failure after the row was written does not add one. - CounterFile::store accepts the log fields of a harvest. For an uploaded file it reads the report type, the period and the Exceptions row from the file header. - CounterLog gains sushi_code_status, decoded_logdetails and a background_job accessor. - The background job passes its id to the harvest and reports the harvest status, shown on the job detail page with a badge per status. Test plan: 1. Run updatedatabase. Open a provider that had import logs before the patch, tab "Import logs": the old rows show status "Success" 2. Run a harvest over two months. On the Import logs tab, check one row per report type and month with the columns Filename, Report type, Period, Status, Details, Import date, Imported by and Job. The job link opens the background job page, which shows a "Harvest status" badge 3. Create a provider with wrong credentials and run a harvest: the job finishes, the Import logs tab shows a row with status "Error", no file name, the exception code and message in Details, and no Download or Delete action. Change the URL to a host that does not exist and run a harvest again: the new row shows status "Retry" with HTTP 500 and the connection error in Details 4. Harvest a month for which the provider has no usage (if the provider allows it): the row shows status "No usage" or "No data" and the job page shows the same badge 5. Use the Status filter of the table: select "Error", then "Success" and check the rows shown 6. Upload a COUNTER file on the Manual upload tab: its row shows the report type and the period read from the file header, status "Success", Download and Delete actions, and no job link 7. Delete the uploaded file: the row disappears 8. Age the harvest jobs so the cleanup can purge them, in koha-mysql: UPDATE background_jobs SET ended_on = DATE_SUB(NOW(), INTERVAL 2 DAY) WHERE type = 'erm_sushi_harvester'; Then run misc/cronjobs/cleanup_database.pl --jobs-days 1 --jobs-type erm_sushi_harvester --confirm. The import log rows stay, with an empty Job column, and the job link is gone 9. prove -l t/db_dependent/Koha/ERM/EUsage/CounterLog.t \ t/db_dependent/Koha/ERM/EUsage/UsageDataProvider.t \ t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t \ t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t \ t/db_dependent/Koha/BackgroundJob/ErmSushiHarvester.t \ t/db_dependent/api/v1/erm_counter_logs.t 10. yarn cypress run --spec t/cypress/integration/ERM/DataProviders_spec.ts Assisted-by: Fable 5.1 (Anthropic) -- You are receiving this mail because: You are watching all bug changes. _______________________________________________ Koha-bugs mailing list -- [email protected] To unsubscribe send an email to [email protected] website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
