https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417
Jonathan Druart <jonathan.dru...@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff --- Comment #30 from Jonathan Druart <jonathan.dru...@bugs.koha-community.org> --- So, getting back to this (one year after the previous try). Please read this comment, the different commit, then try the patchset. I have changed some stuffs, in order to make everybody (hopefully) happy. I removed the dependency to Net::RabbitFoot, in order to not depend on non-packaged stuffs, and I decided to take a slightly different direction. This patch can be considered ready for testing. I have investigated a bit what could be possible, and picked STOMP [1]: """ STOMP is the Simple (or Streaming) Text Orientated Messaging Protocol. STOMP provides an interoperable wire format so that STOMP clients can communicate with any STOMP message broker to provide easy and widespread messaging interoperability among many languages, platforms and brokers. """ There is an actively developed Perl library, Net::Stomp, packaged for debian. I think it could be a very good first step, as we could have a dependency on a small server at first, and sysop could then use more powerful/scalable/etc server depending on their needs. However I sticked to RabbitMQ for the server in my tests, and it worked great. Something else worth to be noted: STOMP does not provide a 'namespace' parameter that can be specified at connection time, unlike Net::RabbitFoot. But there is a 'destination' on send/receive (actually when subscribing), that seems to do the same job. There is a new patch set (see the same remote branch, previous one is pushed to old/20200225_bug_22417), with different follow-ups adding and fixing several things (better error handling , fix batch authority modification, etc.) Test plan: 0. Prerequisite: % apt install rabbitmq-server # server % rabbitmq-plugins enable rabbitmq_stomp # Enagle the STOMP plugin % sudo service rabbitmq-server status # Make sure the RabbitMQ server is running % sudo service rabbitmq-server start # start it if needed % apt install libnet-stomp-perl # install the client lib % cp debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh % cp debian/scripts/koha-worker /usr/bin/ % koha-worker --start kohadev # start the background jobs worker % tail -f /var/log/rabbitmq/* # take a look at the look to understand what's going on % Execute the updatedatabase entry to create the new DB table Then create, or make sure you have a MARC Modification Template, with template_id=1 (like add new field 123$z=foo) 1. To test this patch set you should first understand how the enqueue/consume process work: % perl new_koha_job.pl # Launch it 1+ times => The job must have been enqueued (no error in the output) % perl koha_worker.pl # Will consume the 1+ tasks you enqueued Do not close this worker and start another one in another console, then enqueue more new jobs. Use the following bash loop to enqueue lot of jobs: % for i in {1..100}; do perl new_koha_job.pl; done Enjoy :) (yes it should be fun) 2. Watch the history of the jobs Hit admin/background_jobs.pl (logged in as superlibrarian) This is an interface of the list of pending/running and finished processes 3. Use this whole stuff in a real Koha world: Play with the "Batch record modification" tool (with biblios or authorities) Notice the different in the last (report) step and click on the "View detail of the enqueued job" link Discussions: 1. Please grep TODO and FIXME in the patchset, there are lot to do/fix. 2. Even if we do not go for STOMP I think we could start moving the code from pl to Koha::BackgroundJob::*pm It will be "easy" (nothing is easy, right?) to replace it with something else. 3. With this base we will want to answer different needs: * background jobs * task scheduler * reindex process (?) Going further: 1. Implement Koha::BackgroundJob for the different background jobs 2. Rewrite the task scheduler with Koha::BackgroundJob (1993) 3. Provide a configuration file to prioritize or postpone jobs (for instance no batch modification on Wednesday between 9-17) - Or maybe we won't need that (?) 4. Server-side, there is the possibility to configure a lot of things, like users, destinations, ssl, etc. By default guest/guest has a localhost access only. Code is at https://gitlab.com/joubu/Koha/commits/bug_22417 [1] https://stomp.github.io/ https://stomp.github.io/stomp-specification-1.2.html -- You are receiving this mail because: You are watching all bug changes. _______________________________________________ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/