[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #56 from Jonathan Druart --- Can we dedicate this bug report to the base idea, then create bug report on top of it for experiment? I'd like to avoid noise for testers and keep the comment list easy to follow. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #55 from David Cook --- At Tomas's request, I've attached a patch I've done for a local customization to use RabbitMQ. It's very simple. I don't have any time-based "task scheduling" per se. It's just FIFO. The mq_scheduler.pl script is problematic in a number of ways, but it's just an experiment. Early testing is good. You submit a form on the Koha web UI, it sends a message to RabbitMQ, the mq_scheduler.pl receives the message from RabbitMQ, the mq_scheduler.pl loads a module and runs some code (which in this case also sends an email), and then it marks the message complete/acknowledged in RabbitMQ. This patch does require a person to run "add_vhost" and "set_permissions" manually using rabbitmqctl on the host running RabbitMQ. However, as I'm defining the queues in koha-conf.xml, the root vhost could be used, and the queues could be prefixed with __DB_NAME__. At this point, I'm fairly indifferent to the method. I like vhosts for the logical separation of the messages, but if it's not practical for a package installation, then maybe we should just use the prefixed queues. The systemd service is *very* package installation friendly. We'd need to update the debian/* files accordingly, but it would be so easy. I haven't done that yet, as I'm testing on non-package installations first. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #54 from David Cook --- Created attachment 105543 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=105543=edit [Local Prosentient] Create experimental support for RabbitMQ-based background tasks This patch includes a koha-mq-scheduler service which creates a mq_scheduler.pl daemon, which listens to RabbitMQ queues. This patch also includes an audit-authorities.pl web script, which sends a message to RabbitMQ. The mq_scheduler.pl daemon loads Koha::Prosentient::MQ::AuditAuthorities.pm, and runs a class method which does a background task to audit bib-auth linkages. Koha community work: https://bugs.koha-community.org/bugzilla3//show_bug.cgi?id=22417 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook changed: What|Removed |Added Attachment #105542|1 |0 is patch|| Attachment #105542|0 |1 is obsolete|| -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #53 from David Cook --- Created attachment 105542 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=105542=edit [Prosentient] Create experimental support for RabbitMQ-based background tasks This patch includes a koha-mq-scheduler service which creates a mq_scheduler.pl daemon, which listens to RabbitMQ queues. This patch also includes an audit-authorities.pl web script, which sends a message to RabbitMQ. The mq_scheduler.pl daemon loads Koha::Prosentient::MQ::AuditAuthorities.pm, and runs a class method which does a background task to audit bib-auth linkages. Koha community work: https://bugs.koha-community.org/bugzilla3//show_bug.cgi?id=22417 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #52 from David Cook --- Oh and in the JSON message I specify "class" and "method" (much like the Koha::Plugin::Handler). It also provides "args". So it loads the "class", and runs the "method" with the "args". It's pretty simple. If I were to record results, I'd probably have a "result_callback_method" or something too. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #51 from David Cook --- (In reply to David Cook from comment #50) > We use RabbitMQ for some other projects, and I think I'm going to start > looking at using it with some of our local Koha extensions. > With my own code, I've added a local tool to tools-home.pl, and I have that add a JSON message to RabbitMQ with Net::Stomp. I have a background worker using Net::Stomp to get the JSON message, do the long-running task, and then email a user with the result. It does the bare minimum of what I need it to do, and I'll be looking to roll it out soon as an experimental feature to select Koha instances. Both the web app and daemon worker rely on configuration in koha-conf.xml to know not only the RabbitMQ server address, username, and password, but also what queues to use. (I rather use a YAML file for this, but koha-conf.xml already has caching support, so it's so easy to use.) Things to think about: - Installing RabbitMQ and plugins are easy, but you need enable the stomp plugin and create virtual hosts as the RabbitMQ administrator... which is only possible if they're running on the same machine - Technically, all Koha instances could use the root vhost, and we could prefix the queue names. (I'm going to be running into this issue when I roll out my experimental work more widely.) - Flow control for the background worker. Should it do 1 task at a time, or should it fork off workers? If it forks workers, you can easily exhaust your system's resources if you don't have any artificial limits, but you can also do more work. (Alternatively, we could create the facility to run X background workers, so sysadmins would determine the limits.) - How best to store the results of background tasks? Should it be in 1 central store, or should it be more task specific? Most of those are sysadmin type questions except for the last one. For the last one, I'm thinking task specific might be best, especially when it comes to background tasks done by Koha Plugins. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Jonathan Druart changed: What|Removed |Added Keywords||rel_20_11_target, ||RM_priority -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #50 from David Cook --- We use RabbitMQ for some other projects, and I think I'm going to start looking at using it with some of our local Koha extensions. I don't have a timeline. It's mostly just going to be when I have some free time, but happy to report back on how it goes, lessons learned, etc. Tomas, I'm also interested in using RabbitMQ with Koha plugins. I think I agree now with the general model that you suggested. My koha-plugin-oaipmh-import plugin would be able to make use of RabbitMQ, so I'd be keen to test it out on the plugin side. I suppose the minimum we'd need in Koha is the following: 1) A RabbitMQ server 2) A background Koha worker (which can invoke Koha plugins) The plugin worker could write results to its own result store (ie a plugin controlled database table). A core (ie non plugin) result store is probably a good idea but not an immediate requirement I would think... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #49 from David Cook --- (In reply to Tomás Cohen Arazi from comment #47) > - I would like to know what ideas you might have for maintaining the 'tasks' > catalog, specially when it comes to plugins (i.e. the mapping between > background_job.type and the method that has to be run. I am really interested in this too. > - I expected to see an event-driven implementation of the koha_worker.pl > loop, as messages will arrive and we should react to them instead of > polling. Maybe it is a limitation from Net::STOMP, I saw there's > AnyEvent::STOMP which is event-driven. Maybe the 'blocking and waiting for a > frame' behaviour from Net::STOMP::receive_frame works similarly... worth > checking. As you say, "blocking and waiting for a frame" is event-driven behaviour, so koha_worker.pl is already event-driven. Net::STOMP and AnyEvent::STOMP both use a select-based event loop. (AnyEvent can use other event loops apparently, but you have to load them. Plus the performance gained using a more sophisticated event loop is likely marginal for our use cases.) > I will organise my work so I can spend some hours on redoing this using > AnyEvent::Task::Server and AnyEvent::Task::Client as I wrote a POC a lot of > time ago. > That looks interesting! Can I suggest that you use a TCP socket rather than a Unix socket? For my POE-based task scheduler in Bug 10662 (which was similar to this but included timers for task scheduling) I used a Unix socket and it worked fine, but it meant my scheduler has to be on the same machine; in hindsight, I wish I had used a TCP socket. With a TCP socket, it would make Dockerization easier and could make production deployments easier (depending on how the workers are done). Actually, how do you see AnyEvent::Task fitting in with RabbitMQ or task scheduling? The AnyEvent::Task::Server would sit in the background... would your koha_worker.pl be the RabbitMQ consumer and AnyEvent::Task::Client? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #48 from David Cook --- (In reply to Tomás Cohen Arazi from comment #47) > I'm impressed by the results, Jonathan. I really sorry I didn't spend time > on this before. It deserved it for sure. > Ditto. I would love to see more interest in this one, as it could be such a valuable addition to Koha. I don't know why more people don't seem interested. Not sure if they're just uninterested because they're not familiar with the technologies. > I will probably repeat myself, but in my opinion what we need is: > > - A task_queue table > - A daemon that looks for things to do, and spawns children processes to do > the tasks, mark them as completed or failed, handle retries, etc > This is similar to the model that I've used in the past (for Bug 10662 and some of my fun projects), and it has pros and cons. Personally, I am interested in a model where we create a task scheduler daemon that Koha connects to via a HTTP API, and the task scheduler daemon publishes to the message broker when it's time to run a task. It has pros and cons too: FORKING: - One pro is that it closely couples the task scheduler to Koha, which makes it very easy to *do things*. - One con is that it can only use 1 machine's resources, which reduces its availability and scalability. It also means you're probably running 1 task scheduler per Koha instance, which could increase management/maintenance complexity. - One pro is that it's very easy to increase/decrease limits on forking child processes. - One pro is that it would be easier to use plugins to do background jobs, as the task scheduler would just need to load the code. WORKERS: - One pro is that it loosely coupled the task scheduler to Koha, which means it is a discrete modular unit of code that can be developed and tested independently of the rest of the Koha codebase. - One con is that being loosely coupled makes it harder to do things. It means using more APIs instead of giving direct database access and internal (ie Perl) API access to workers. - One pro is that a sysadmin has a lot of control for increasing/decreasing the number of worker processes, and where they run. (This lends itself to containerization.) - One pro is that it means workers can be written in any programming language. Tasks which require high efficiency could have workers written in a more high performing language. - One con is that it increases complexity. Managing worker processes is harder than forking new child processes. Supervisord could be useful for this though. This is a solved problem. - One con is that plugins wouldn't be able to introduce worker processes. (Slowly, in my free time, I'm developing a Koha plugin which will connect to a Golang-based task scheduler using a HTTP API which runs OAI-PMH harvest jobs using goroutines. The downside is that a sysadmin would have to manage the Golang task scheduler. It wouldn't work just out of the box after loading the Koha plugin.) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #47 from Tomás Cohen Arazi --- I'm impressed by the results, Jonathan. I really sorry I didn't spend time on this before. It deserved it for sure. I will probably repeat myself, but in my opinion what we need is: - A task_queue table - A daemon that looks for things to do, and spawns children processes to do the tasks, mark them as completed or failed, handle retries, etc That's what basically Minion does. This implementation: - Sets a message broker - Has a table to keep track of tasks - Sends a message for each task that is enqueued - Has a daemon that looks for those messages and runs the task What we are doing is modelling a task queue with a message queue, from which we use (probably) its scheduling features, and the 'retry' logic perhaps? I'm not in a situation in which I can provide an alternative right now, so I wouldn't want to block this right now. So will try to add my constructive two cents: - I would like to know what ideas you might have for maintaining the 'tasks' catalog, specially when it comes to plugins (i.e. the mapping between background_job.type and the method that has to be run. - I expected to see an event-driven implementation of the koha_worker.pl loop, as messages will arrive and we should react to them instead of polling. Maybe it is a limitation from Net::STOMP, I saw there's AnyEvent::STOMP which is event-driven. Maybe the 'blocking and waiting for a frame' behaviour from Net::STOMP::receive_frame works similarly... worth checking. - I would like to see OO in Koha::BackgroundJob but this is minor abviously I will organise my work so I can spend some hours on redoing this using AnyEvent::Task::Server and AnyEvent::Task::Client as I wrote a POC a lot of time ago. GOOD JOB Jonathan, it is obvious you spent lots of time on this and you put the bar very high. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Tomás Cohen Arazi changed: What|Removed |Added Severity|enhancement |new feature -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Tomás Cohen Arazi changed: What|Removed |Added Version|unspecified |master CC||k...@bywatersolutions.com, ||n...@bywatersolutions.com --- Comment #46 from Tomás Cohen Arazi --- Clap Clap Clap -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #45 from David Cook --- Thinking out my plugin idea for Bug 10662 made me think how useful it would be to have RabbitMQ support built into Koha (via STOMP), but it made me wonder how you'd provide the workers in the plugin. I suppose a person could just bundle it in with the plugin code, but the plugin code is "web" code. Might not matter depending on how the web server is set up I suppose. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #44 from David Cook --- I've also been thinking about the separate needs for a message queue and a task scheduler. For instance, "Bug 1993 - Task Scheduler Needs Re-write" and an OAI-PMH harvester would both need a task scheduler to make sure tasks run at schedule times. However, a Staged MARC Import doesn't need to be scheduled. It just needs to be run ASAP in the background. (Same case for batch modifications, label creation, and many other long-running tasks.) That being said, for scheduled reports (like in "Bug 1993 - Task Scheduler Needs Re-write"), the scheduled task itself might be to enqueue a message into the message queue. (For instance, let's say 20 librarians schedule their SQL reports to run at 6pm. At 6pm, those 20 SQL reports should be put into a queue, and then 1+ report workers can do the work and record the results. They then queue an email job, and emails go out with the results.) (Arguably email processing could be a scheduled task, as you may or may not want emails going out from the library in the middle of the night.) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #43 from David Cook --- I am keen to get Koha using RabbitMQ, so I'm wondering a bit about the shortest path for us to do that. I am tempted to start making Koha plugins that use RabbitMQ as an experiment. That might make it easier to decide on a tried and tested way to proceed with master Koha code. Plus, plugins are well suited to batch/long-running work that RabbitMQ would help with. (Honestly, we have some local customizations that I want to change to use RabbitMQ anyway, so I think that I'm heading in that direction with or without the community anyway, but I'd like to share that work and knowledge.) Why not add Net::Stomp as an optional dependency for Koha? And then maybe make some experimental features that can be switched on using system preferences? (That is to say, it might be better to experiment for a bit, rather than trying to completely replace existing components.) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #42 from David Cook --- I am tempted to try re-engineering https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 using STOMP and RabbitMQ, as I would really like to use a standard message queue rather than my bespoke implementation. I would also be interested in using plugins to provide a Koha CRUD web interface for managing STOMP messages for specific types of tasks. For instance, I could provide a plugin for sending add/start/stop/update/remove messages to an OAI-PMH harvester. As the sysadmin, I'd need to run up the harvester in the background first, but that would be OK. It would be more so for a proof of concept than anything at first. Anyway definitely bed time now... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #41 from David Cook --- git diff ff4c85109f043c4086c9348e5e88638cc5a7ebd5~1..HEAD TODOs: 1. Agree raising exceptions is better than returning null 2. A new "manage_background_jobs" permission might make sense for managing supervisor permissions, but I figure whoever enqueued the job should be able to check on its progress, so this might not make sense. We should manage background job permissions based on their other permissions, although that gets tricky. I suppose we could pass the background job type, validate it against a list (and translate it to an existing permission), and then put it into flagsrequired. If there is no background job type passed, then we assume it's all, and then we could use "manage_background_jobs" as the permission which would allow a supervisor/superlibrarian to view all jobs at once? Alternatively, admin/background_jobs.tt is for supervisors/superlibrarians only, and each tool implements its own interface for tracking tasks. That might be necessary in any case, as the existing functionality isn't very librarian friendly. While "Background Jobs" make sense to developers, I don't think librarians will necessarily understand the concept, nor should they need to. FIXMEs: Too many to comment on right now, plus would be easier to comment on if you uploaded a patchset to Bugzilla. I think we might have too much specific/tightly-coupled logic in Koha/BackgroundJob.pm. I notice too there is some "enqueue" copy/paste code between Koha/BackgroundJob/BatchUpdateBiblio.pm and Koha/BackgroundJob/BatchUpdateAuthority.pm, which could be put into a base/parent class. I think it would make sense to put the "process" code into something like "Koha/Worker/BatchUpdateBiblio.pm", and to use a "Koha/Worker/Base.pm" type class to handle the messaging aspect for the workers. I could provide more feedback but it is late and I need to get up early. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #40 from David Cook --- Ok I'm finally trying this out! (I should be Netflixing or chilling or something, but I am really keen to give this a go, especially after a day of talking about message queues for other projects. Plus I'm always saying we need to modernize, so I really want to help out Jonathan with all his efforts on this one, and show that it's not for nothing.) My steps: 0a. cd ~/git/koha-testing-docker/ 0b. sh run.sh (my own script to for invoking docker-compose with a custom file) 1. winpty docker exec -it koha_koha_1 bash 2. cd koha 3. git remote add joubu https://gitlab.com/joubu/Koha.git 4. git fetch joubu 5. git checkout -b rabbitmq joubu/bug_22417 #The following could probably be handled using https://hub.docker.com/_/rabbitmq 6. apt install rabbitmq-server 7. rabbitmq-plugins enable rabbitmq_stomp 8. service rabbitmq-server start 9. service rabbitmq-server status 10. apt install libnet-stomp-perl 11. cp debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh 12. cp debian/scripts/koha-worker /usr/bin/ 13. koha-worker --start kohadev 14. koha-shell kohadev -c installer/data/mysql/updatedatabase.pl 15. restart_all 16. Log into http://localhost:8081/ 17. Go to Tools > MARC modification templates 18. Make new template 18b. Name: test 19. Make new action for "test" template 19b. Add new field "856" "u" "https://koha-community.org; 19c. Description: "RabbitMQ test" 20. koha-shell kohadev 21. perl new_koha_job.pl Output: [x] Enqueuing BatchUpdateBiblio mmtid=1 with biblionumber=1,2,3 22. exit 23. Go to http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 24. Confirm that the job has completed 25. Go to http://localhost:8081/cgi-bin/koha/tools/batch_record_modification.pl 26. Enter list of record numbers: 1, 2, 3, 4, 5 27. Select "test" template 28. Click "Continue" 29. Click "Modify selected records" 30. See message "The job has been enqueued! It will be processed as soon as possible (FIXME - well, it could depend on a config?)" 31. Click "View detail of the enqueued job" 32. Confirm job has status of "finished" 33. Click "Return to the job list" (or "New batch record modification") Great work, Jonathan! The test works beautifully. I'll take a peek at the code now to look for some of those FIXME and TODO messages you mention. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #39 from David Cook --- (As a side note, I am looking at packaging (as RPMs) newer versions of DBD::Pg and DBI to provide access to new PostgreSQL versions for older OSes for a different project, and it doesn't seem too challenging.) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #38 from David Cook --- (In reply to Julian Maurice from comment #35) > STOMP is very simple indeed. But combined with RabbitMQ it becomes a > powerful solution for dispatching messages. And that's not a bad thing of > course, but I think it adds a useless layer of complexity in our case. I am willing to try both Minion and RabbitMQ (and I keep telling myself I will try them soon in my personal time since I'm staying home due to the pandemic). But I want to say first that I don't think we'd be adding a useless layer of complexity with RabbitMQ. Provided the implementation is well-architected, I think message queues are an integral part of scaling up an application and improving interoperability. Using a language agnostic protocol like STOMP, it could be easier to implement higher performance workers in other languages. For instance, maybe we'd want to implement some workers in Golang, which makes concurrent programming very easy with its implementation of user threads as goroutines balanced against kernel threads depending on resource demands. For my work on a OAI-PMH harvester, I have been thinking of rewriting it in something like Golang instead of Perl, and then hooking into Koha using a standard message queue (instead of my bespoke POE-based implementation) and/or APIs. (Honestly though my Perl version is quite high performance in terms of downloads but the bottleneck is importing into Koha. That's still slower than I'd like. I occasionally think about how to optimally implement a scalable high performance MARCXML record importer. But that's a whole other topic really.) That said, RabbitMQ is a new dependency, and I think our collective fear of new dependencies has kept us from implementing a message queue previously. I think we've arrived at a point where we *have* to have some kind of job/message queue though. Anyway, sorry for the ramble. It's just a topic in which I have a strong interest. I'm going to put it at the top of my personal to try pile. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #37 from Julian Maurice --- (In reply to Jonathan Druart from comment #36) > Does it allow additional data? Yes, there is a 'note' field for this purpose https://mojolicious.org/perldoc/Minion/Job#note > Will you manage to write the dashboard I provided with my patches? > When I did my different tries bug 15032 it was not possible (not > Minion). Here are some screenshots of the admin UI packaged with Minion: https://snipboard.io/n0O6lf.jpg https://snipboard.io/QNJrLC.jpg https://snipboard.io/tiq5dI.jpg -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #36 from Jonathan Druart --- (In reply to Julian Maurice from comment #34) > (In reply to Jonathan Druart from comment #33) > > So you need an additional DB. 2 tables to store the jobs (one for our data, > > to keep track of them, one internal for minion). > > What ? No. You can use the same database, and you do not have to keep track > of jobs manually. Minion does that for you. Does it allow additional data? Will you manage to write the dashboard I provided with my patches? When I did my different tries bug 15032 it was not possible (not Minion). -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #35 from Julian Maurice --- (In reply to Jonathan Druart from comment #33) > I do not think STOMP is "very powerful", I picked it for its simplicity. > Having the RabbitMQ option will let sysadmins decide the flexibility they > want. STOMP is very simple indeed. But combined with RabbitMQ it becomes a powerful solution for dispatching messages. And that's not a bad thing of course, but I think it adds a useless layer of complexity in our case. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #34 from Julian Maurice --- (In reply to Jonathan Druart from comment #33) > So you need an additional DB. 2 tables to store the jobs (one for our data, > to keep track of them, one internal for minion). What ? No. You can use the same database, and you do not have to keep track of jobs manually. Minion does that for you. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Jonathan Druart changed: What|Removed |Added Status|Needs Signoff |In Discussion --- Comment #33 from Jonathan Druart --- (In reply to Julian Maurice from comment #32) > Hi, > > I played again, 1 year later, with this patchset and also with Minion. > > The STOMP && RabbitMQ solution looks very powerful, but I think that it is a > little bit too much for our needs. > > I made a POC with Minion for comparison. > https://gitlab.com/jajm/Koha/-/commits/bug/22417-minion > There are 2 commits, there's a test plan in each commit message. So you need an additional DB. 2 tables to store the jobs (one for our data, to keep track of them, one internal for minion). I do not think STOMP is "very powerful", I picked it for its simplicity. Having the RabbitMQ option will let sysadmins decide the flexibility they want. Back to In Discussion then. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #32 from Julian Maurice --- Hi, I played again, 1 year later, with this patchset and also with Minion. The STOMP && RabbitMQ solution looks very powerful, but I think that it is a little bit too much for our needs. I made a POC with Minion for comparison. https://gitlab.com/jajm/Koha/-/commits/bug/22417-minion There are 2 commits, there's a test plan in each commit message. About the packaging problems mentioned in comment 1 > I first tried Minion but found problematic issues for us (Koha): > - The mysql backend requide Mojo::mysql which requires DBD::mysql 4.042 > However Jessie has 4.028 and Stretch 4.041 This is still true, but buster is out for a while now and has a more recent version of DBD::mysql, so it should not be a problem to package Mojo::mysql and Minion::Backend::mysql. Older distributions could use the SQLite backend which works well too. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #31 from David Cook --- Going to try to make some time to try this out next week. For the OAI-PMH harvester, I'm going to want the user to be able to enqueue messages for starting/stopping downloading. Downloader workers will need to be able to enqueue messages for other downloader workers (for recurring tasks with slightly altered parameters each time). Downloader workers will also need to be able to enqueue messages for import workers. I'll think about those logistics later, but it should be doable with a few topics/queues in the message broker. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417
Jonathan Druart changed:
What|Removed |Added
Status|ASSIGNED|Needs Signoff
--- Comment #30 from Jonathan Druart
---
So, getting back to this (one year after the previous try).
Please read this comment, the different commit, then try the patchset.
I have changed some stuffs, in order to make everybody (hopefully) happy.
I removed the dependency to Net::RabbitFoot, in order to not depend on
non-packaged stuffs, and I decided to take a slightly different direction.
This patch can be considered ready for testing.
I have investigated a bit what could be possible, and picked STOMP [1]:
"""
STOMP is the Simple (or Streaming) Text Orientated Messaging Protocol.
STOMP provides an interoperable wire format so that STOMP clients can
communicate with any STOMP message broker to provide easy and widespread
messaging interoperability among many languages, platforms and brokers.
"""
There is an actively developed Perl library, Net::Stomp, packaged for debian.
I think it could be a very good first step, as we could have a dependency on a
small server at first, and sysop could then use more powerful/scalable/etc
server depending on their needs.
However I sticked to RabbitMQ for the server in my tests, and it worked great.
Something else worth to be noted: STOMP does not provide a 'namespace'
parameter that can be specified at connection time, unlike Net::RabbitFoot. But
there is a 'destination' on send/receive (actually when subscribing), that
seems to do the same job.
There is a new patch set (see the same remote branch, previous one is pushed to
old/20200225_bug_22417), with different follow-ups adding and fixing several
things (better error handling , fix batch authority modification, etc.)
Test plan:
0. Prerequisite:
% apt install rabbitmq-server # server
% rabbitmq-plugins enable rabbitmq_stomp # Enagle the STOMP plugin
% sudo service rabbitmq-server status # Make sure the RabbitMQ server is
running
% sudo service rabbitmq-server start # start it if needed
% apt install libnet-stomp-perl # install the client lib
% cp debian/scripts/koha-functions.sh /usr/share/koha/bin/koha-functions.sh
% cp debian/scripts/koha-worker /usr/bin/
% koha-worker --start kohadev # start the background jobs worker
% tail -f /var/log/rabbitmq/* # take a look at the look to understand what's
going on
% Execute the updatedatabase entry to create the new DB table
Then create, or make sure you have a MARC Modification Template, with
template_id=1 (like add new field 123$z=foo)
1. To test this patch set you should first understand how the enqueue/consume
process work:
% perl new_koha_job.pl # Launch it 1+ times
=> The job must have been enqueued (no error in the output)
% perl koha_worker.pl # Will consume the 1+ tasks you enqueued
Do not close this worker and start another one in another console, then
enqueue more new jobs.
Use the following bash loop to enqueue lot of jobs:
% for i in {1..100}; do perl new_koha_job.pl; done
Enjoy :)
(yes it should be fun)
2. Watch the history of the jobs
Hit admin/background_jobs.pl (logged in as superlibrarian)
This is an interface of the list of pending/running and finished processes
3. Use this whole stuff in a real Koha world:
Play with the "Batch record modification" tool (with biblios or authorities)
Notice the different in the last (report) step and click on the "View detail of
the enqueued job" link
Discussions:
1. Please grep TODO and FIXME in the patchset, there are lot to do/fix.
2. Even if we do not go for STOMP I think we could start moving the code from
pl to Koha::BackgroundJob::*pm
It will be "easy" (nothing is easy, right?) to replace it with something else.
3. With this base we will want to answer different needs:
* background jobs
* task scheduler
* reindex process (?)
Going further:
1. Implement Koha::BackgroundJob for the different background jobs
2. Rewrite the task scheduler with Koha::BackgroundJob (1993)
3. Provide a configuration file to prioritize or postpone jobs (for instance no
batch modification on Wednesday between 9-17) - Or maybe we won't need that (?)
4. Server-side, there is the possibility to configure a lot of things, like
users, destinations, ssl, etc. By default guest/guest has a localhost access
only.
Code is at https://gitlab.com/joubu/Koha/commits/bug_22417
[1] https://stomp.github.io/
https://stomp.github.io/stomp-specification-1.2.html
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #29 from David Cook --- I won't be at the Marseille hackfest, but maybe some folk will think about working on this there next month? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Jonathan Druart changed: What|Removed |Added Status|In Discussion |ASSIGNED -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Ray Delahunty changed: What|Removed |Added CC|r.delahu...@arts.ac.uk | -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #28 from Michal Denar --- Hello, Rabbit is pretty useful for Koha in my eyes. Now we can fight with some limitation because message_que is "one piece" If some message is in the que and waiting for send isn't possible delete just one or group of them. So fragmentation based on RabbitMQ can be great solution. But this bring many new possibillities and better services to borrowers and stuff. I vote for this implementation. May I help with testing or something else? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Michal Denar changed: What|Removed |Added CC||blac...@gmail.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Ray Delahunty changed: What|Removed |Added CC||r.delahu...@arts.ac.uk -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Magnus Enger changed: What|Removed |Added CC||mag...@libriotech.no -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #27 from David Cook --- (In reply to David Cook from comment #26) > Anyway, mostly just playing devil's advocate and trying to think of edge > cases. I suppose the advantage of the cronjob is that system administrators would control the timing of it. I suppose there could be a warning on the page saying that the timing of tasks scheduled for the future would be up to the discretion of the system administrator? Lately, I've been wondering how we can modernize Koha, so that it can run in large distributed environments... but also on low budget single server systems. Being all things to all people... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #26 from David Cook --- (In reply to Juan Romay Sieira from comment #25) > * Use version 3.5.3 or later of RabbitMQ. The one I have installed with > Debian Jessie is 3.3.5. From version 3.5.3 you can use a plugin for delayed > messages, so that these can be consumed in the future, and not immediately, > which is how RabbitMQ works. I do not know how Net::RabbitFoot will behave > in this case ... > This sounds like it's probably not an option due to the version issue. > * Use an intermediate table (or the background_jobs table, with a new column > called exec_on) for future messages, and have a producer (cronjob) to send > the messages to RabbitMQ at the time they need to be consumed. > If you're going to use a cronjob as a producer for future messages, maybe it's worthwhile to use a cronjob as a producer for all messages. That way there's just 1 method for enqueuing messages. My only reluctance to use a cronjob is that it relies on a system administrator setting it up properly, and it's challenging to know from the web interface if it has been set up correctly, so the user experience could suffer greatly. How would users know that the cronjob is running and their task will actually be run? As a result, it seems better to run a task scheduler service, but then that gets us back to using existing tools like Minion, Celery, etc. I suppose we could write our own task scheduler though. That reminds me of something that Fridolin said to me at Kohacon18 though. From a vendor/system administrator perspective, we don't necessarily want libraries to be in-charge of their own scheduling. For instance, say a vendor is running 30 Koha instances with a shared database server, and the librarians at each of the 30 Koha libraries all schedule a very intensive report to run at 7pm. Or say they schedule a task that does API calls, and then an upstream API server rejects all the calls after the 10th due to rate limiting by domain. I mention this because I already wrote my own task queue/task scheduler (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html=10662=85224), and Fridolin at BibLibre pointed out that since each instance ran their own task scheduler which could schedule a task for 7pm, they could cause these problems. I suppose in this case the timers would just be for the enqueuing of tasks. The dequeuing and execution would still be handled by RabbitMQ, and I suppose if you ran with a low number of workers you'd be less likely to overwhelm your systems. And if you needed better performance, you could add workers and add database servers for the report scenario. The rate limiting scenario is trickier though. I don't know how to solve that one. My only ideas for that involve complex home-baked solutions. I don't know the best practice for that one. Anyway, mostly just playing devil's advocate and trying to think of edge cases. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Juan Romay Sieira changed: What|Removed |Added CC||juan.sie...@xercode.es --- Comment #25 from Juan Romay Sieira --- Thanks Jonathan for the great job done. I have been testing the development, it works correctly and it seems to me a very good idea to use RabbitMQ as a queue manager. As I had told you by email, we have to do a development focused on the reports, so that they can be executed regardless of AT, and also can be scheduled reports that require user parameters at the time of execution. Because the reports should be able to be programmed, the way to work with RabbitMQ would be through two possibilities: * Use version 3.5.3 or later of RabbitMQ. The one I have installed with Debian Jessie is 3.3.5. From version 3.5.3 you can use a plugin for delayed messages, so that these can be consumed in the future, and not immediately, which is how RabbitMQ works. I do not know how Net::RabbitFoot will behave in this case ... * Use an intermediate table (or the background_jobs table, with a new column called exec_on) for future messages, and have a producer (cronjob) to send the messages to RabbitMQ at the time they need to be consumed. If nobody raises objections to the use of RabbitMQ, I can investigate the above options, and make a POC for scheduled reports. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #24 from David Cook --- (In reply to Jonathan Druart from comment #23) > But then the job will be in the DB table, so we will not lose it, right? If we're worried about losing messages, why not look at RabbitMQ persistence? https://www.rabbitmq.com/persistence-conf.html -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #23 from Jonathan Druart --- But then the job will be in the DB table, so we will not lose it, right? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #22 from Julian Maurice --- (In reply to Jonathan Druart from comment #21) > (In reply to Julian Maurice from comment #20) > > I'm a little concerned about the potential loss of jobs with the RabbitMQ > > option. Is there any way to avoid this problem ? > > Using Monit to make sure the service is always up? :) > > But indeed we should return an error if the server has not been reached, and > so the job not enqueued. That solves one problem, but not the other: when RabbitMQ already has enqueued jobs but stops before koha_worker.pl had the chance to process them. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #21 from Jonathan Druart --- (In reply to Julian Maurice from comment #20) > I'm a little concerned about the potential loss of jobs with the RabbitMQ > option. Is there any way to avoid this problem ? Using Monit to make sure the service is always up? :) But indeed we should return an error if the server has not been reached, and so the job not enqueued. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #20 from Julian Maurice --- I played a little with this patchset, and also with Minion. Some remarks: - Net::RabbitFoot says it's compatible with version 0-8 of the AMQP specification. RabbitMQ website says it implements version 0-9-1. Are we sure these two are compatible ? - Net::RabbitFoot is quite easy to package for Debian stretch/stable (all dependencies are already packaged - I did not try on oldstable) - If rabbitmq-server is not started, new_koha_job.pl and koha_worker.pl do not work. Is there a way to enqueue job when rabbitmq-server is not started ? - If koha_worker.pl is not started, and I enqueue jobs, and restart rabbitmq-server and then start koha_worker.pl, the worker does nothing. Potential loss of jobs here ? Now for the Minion part, - the mysql backend requires a recent version of Minion (which requires a more recent version of Mojolicious than the one packaged in debian.k-c.org). It might be complicated to package. - the sqlite backend however have some versions that can be packaged and can work with our version of Mojolicious (requires packaging Minion 6.04, Mojo::SQLite 2.002, and Minion::Backend::SQLite 0.009) - it works similarly to the rabbitmq code (a worker that run continuously, some code to enqueue jobs, where all jobs are stored in database) but doesn't require to have something (like rabbitmq-server) listening all the time to work. The worker just fetches queued jobs from its backend when it starts (and also regularly while it's running) - from what I have seen, it requires less code (see for example: https://mojolicious.io/blog/2018/12/10/minion-stands-alone/) - workers can be "paused" (meaning they won't accept new jobs, but active jobs will continue to run) I'm a little concerned about the potential loss of jobs with the RabbitMQ option. Is there any way to avoid this problem ? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #19 from David Cook --- I was thinking about this work this morning. I'm really busy at the moment, but still think this is a good thing to be working on. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Fridolin SOMERS changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=22481 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #18 from Jonathan Druart --- TODO, remove that: Bug 15032: UpdateDB entry +# $dbh->do( "ALTER TABLE biblio ADD COLUMN badtaste int" ); Bug 22417: Add the ability to cancel a job +sleep(5); :) -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #17 from Jonathan Druart --- (In reply to David Cook from comment #15) > If I understand correctly, each Koha instance would get its own RabbitMQ > "vhost", and within each "vhost" there would be named queues (which at the > moment are hard-coded but could be configurable I suppose). I imaged one koha-worker script per Koha instance, with as well one vhost. The queues would be named 'batch_record_modification', 'manage-marc-import', 'batch_item_modification', etc. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #16 from David Cook --- (In reply to Tomás Cohen Arazi from comment #12) > My only doubt is if we should really model the task queue with a message > queue. I tried the same Jonathan tried but using AnyEvent::Task and the > results where similar, and it only required a unix socket to communicate > client and server processes. My suggestion was to use zeromq. But lets see > how this goes. Specially in the integration step. Yeah, for #10662, I used POE::Component::JobQueue and a Unix socket. It was very effective, although my bespoke implementation had a number of limitations. I figured allowing TCP sockets also might be a good idea just to allow for more distributed architectures. When I was working on #10662, I was considering both RabbitMQ and ZeroMQ, but ended up doing my own job queue. I'm not necessarily sold on RabbitMQ yet, but I figure if Jonathan is willing to do it, I'm happy to see how this goes too. I'm not willing to put in the time to make it happen :/. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417
--- Comment #15 from David Cook ---
(In reply to Jonathan Druart from comment #13)
> (In reply to Tomás Cohen Arazi from comment #12)
> > I agree with David that this of course would require a single RabbitMQ
> > server so questions: Is there a concept of namespaces? can any Koha instance
> > access other instance's messages?
>
> I really would like you guys to apply, read and try the patches, read the
> code, then discuss :)
> There are a lot of FIXME and TODO.
>
> This one answers your question:
>
> 16 sub connect {
> 17 my ( $self );
> 18 my $conn = Net::RabbitFoot->new()->load_xml_spec()->connect(
> 19 host => 'localhost', # TODO Move this to KOHA_CONF
> 20 port => 5672,
> 21 user => 'guest',
> 22 pass => 'guest',
> 23 vhost => '/',
> 24 );
> 25
> 26 return $conn;
> 27 }
>
> So different vhost, user and pass can be used.
I haven't tried the patches but I have read your code, and that's why I've
voiced my concerns about multi-tenancy. There wasn't enough information there
to understand, so that's why I asked the questions.
Your reply almost answers the question, but not quite, so now I'm just reading
about RabbitMQ.
If I understand correctly, each Koha instance would get its own RabbitMQ
"vhost", and within each "vhost" there would be named queues (which at the
moment are hard-coded but could be configurable I suppose).
Cool. That removes my concerns about multi-tenancy. Thanks for that, Jonathan.
(Maybe in your example you could use something like "vhost => '/kohademo'" or
maybe include some code comments just so it's more obvious how it works.)
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #14 from Jonathan Druart --- There is also: "Going further: 2. Provide a koha-worker script to consume the jobs" So one worker per instance. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417
--- Comment #13 from Jonathan Druart
---
(In reply to Tomás Cohen Arazi from comment #12)
> I agree with David that this of course would require a single RabbitMQ
> server so questions: Is there a concept of namespaces? can any Koha instance
> access other instance's messages?
I really would like you guys to apply, read and try the patches, read the code,
then discuss :)
There are a lot of FIXME and TODO.
This one answers your question:
16 sub connect {
17 my ( $self );
18 my $conn = Net::RabbitFoot->new()->load_xml_spec()->connect(
19 host => 'localhost', # TODO Move this to KOHA_CONF
20 port => 5672,
21 user => 'guest',
22 pass => 'guest',
23 vhost => '/',
24 );
25
26 return $conn;
27 }
So different vhost, user and pass can be used.
--
You are receiving this mail because:
You are watching all bug changes.
___
Koha-bugs mailing list
Koha-bugs@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #12 from Tomás Cohen Arazi --- (In reply to Marcel de Rooy from comment #11) > (In reply to Katrin Fischer from comment #10) > > (In reply to Marcel de Rooy from comment #9) > > > Do we really need the background jobs ? Or should we refactor these > > > scripts > > > in the direction of AJAX API calls in JavaScript ? > > > > I am not sure if this would also allow reimplementing the scheduler with > > scheduled reports? In my understanding some jobs might not be triggered by > > the GUI. > > OK That's true. I was focused on the forking scripts. IMHO, in either case we need a way to throttle concurrency and server loads. So a task queue is needed. My only doubt is if we should really model the task queue with a message queue. I tried the same Jonathan tried but using AnyEvent::Task and the results where similar, and it only required a unix socket to communicate client and server processes. My suggestion was to use zeromq. But lets see how this goes. Specially in the integration step. I agree with David that this of course would require a single RabbitMQ server so questions: Is there a concept of namespaces? can any Koha instance access other instance's messages? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #11 from Marcel de Rooy --- (In reply to Katrin Fischer from comment #10) > (In reply to Marcel de Rooy from comment #9) > > Do we really need the background jobs ? Or should we refactor these scripts > > in the direction of AJAX API calls in JavaScript ? > > I am not sure if this would also allow reimplementing the scheduler with > scheduled reports? In my understanding some jobs might not be triggered by > the GUI. OK That's true. I was focused on the forking scripts. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Katrin Fischer changed: What|Removed |Added CC||katrin.fisc...@bsz-bw.de --- Comment #10 from Katrin Fischer --- (In reply to Marcel de Rooy from comment #9) > Do we really need the background jobs ? Or should we refactor these scripts > in the direction of AJAX API calls in JavaScript ? I am not sure if this would also allow reimplementing the scheduler with scheduled reports? In my understanding some jobs might not be triggered by the GUI. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Marcel de Rooy changed: What|Removed |Added CC||m.de.r...@rijksmuseum.nl --- Comment #9 from Marcel de Rooy --- Do we really need the background jobs ? Or should we refactor these scripts in the direction of AJAX API calls in JavaScript ? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #8 from David Cook --- (In reply to Josef Moravec from comment #6) > I think just to polish this functionality would be great start and big > progress from what we have now. Actually, I think that's a really good point. I think in the community we sometimes have a tendency to debate/discuss things to death... which stifles innovation. I think it could be good to get something polished into Koha (maybe just 1 background job using RabbitMQ to start) and then if that is working well then we add more background jobs. I suppose it might be more efficient to have it fully specified before implementing in Koha rather than doing iterative development but... -- Oh, but I think my concerns about multi-tenant environments are important. At a glance, this implementation seems like it would only allow 1 Koha instance talk to 1 RabbitMQ instance. But I think in practice there are many people who run multiple Koha instances on the same server and it would be good if they could share 1 RabbitMQ instance. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #7 from David Cook --- (In reply to Jonathan Druart from comment #5) > I have added a commit to add the ability to cancel a job. Thanks, Jonathan! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Josef Moravec changed: What|Removed |Added CC||josef.mora...@gmail.com --- Comment #6 from Josef Moravec --- I just tried it and it works like a charm! Great job Jonathan! I think just to polish this functionality would be great start and big progress from what we have now. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #5 from Jonathan Druart --- I have added a commit to add the ability to cancel a job. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Julian Maurice changed: What|Removed |Added CC||julian.maur...@biblibre.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 claire.hernan...@biblibre.com changed: What|Removed |Added CC||claire.hernandez@biblibre.c ||om -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #4 from David Cook --- (In reply to Jonathan Druart from comment #1) > Discussions: > > 2. Even if we do not go for RabbitMQ I think we could start moving the code > from pl to Koha::BackgroundJob::*pm > It will be "easy" (nothing is easy, right?) to replace it with something > else. I totally agree; it would nice to be as message queue implementation agnostic as possible. > Going further: > 1. Implement Koha::BackgroundJob for the different background jobs > 2. Provide a koha-worker script to consume to jobs > 3. Rewrite the task scheduler with Koha::BackgroundJob (1993) It's been a couple years since I looked at RabbitMQ. Is there a way to cancel/remove/delete a message once it is enqueued? I think this would be useful for a task scheduler. I built that into the POE-based task scheduler I built for https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662. (Actually, I also built-in the ability for in-progress jobs to be canceled in the event a big batch job was started and a person realized it was a mistake.) > 4. Provide a configuration file to prioritize or postpone jobs (for instance > no batch modification on Wednesday between 9-17) - Or maybe we won't need > that (?) Fridolin mentioned something like this to me when I was talking about https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 with him at Kohacon18 in Portland. He pointed out that for a multi-tenant server you might not want all the different Koha instances on the server to run their background jobs at the same time due to restraints like performance, rate-limiting, connection limits, etc. Off the top of my head, I don't have a solution for this one though, although in the case of https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 I use separate task schedulers for every Koha instance on the server. In this case, we could potentially use 1 RabbitMQ server, and X number of workers... although we'd have to have a way of setting the KOHA_CONF for each task. That might be the best way to handle multi-tenant scenarios. Alternatively, I think we'd need to have unique queues for each Koha instance and workers for those unique queues. Requires a bit of thought in any case. > 5. What would be your needs? > Ideally, it would be nice to replace my home-made task scheduler in https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662 with this one. I like that mine can add/start/pause/stop/remove tasks, but I haven't seen a mainstream task scheduler that is capable of doing that, so that might be too difficult to do here. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 David Cook changed: What|Removed |Added CC||dc...@prosentient.com.au --- Comment #3 from David Cook --- I've taken a brief look at the code and the screenshots and it's looking good. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Bob Birchall changed: What|Removed |Added CC||b...@calyx.net.au -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Tomás Cohen Arazi changed: What|Removed |Added CC||tomasco...@gmail.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 --- Comment #2 from Jonathan Druart --- And the screenshots: https://snag.gy/WVeqDf.jpg What you get once the job has been enqueued https://snag.gy/1GgnN5.jpg Detail of a non started job https://snag.gy/ySnlaF.jpg Detail of a finished job https://snag.gy/s5dF7n.jpg List of jobs -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Jonathan Druart changed: What|Removed |Added Status|ASSIGNED|In Discussion --- Comment #1 from Jonathan Druart --- I first tried Minion but found problematic issues for us (Koha): - The mysql backend requide Mojo::mysql which requires DBD::mysql 4.042 However Jessie has 4.028 and Stretch 4.041 - We could have used the Postgres backend, but we would have require MySQL/MariaDB and Postgres, which is not acceptable. I then gave a try to RabbitMQ (https://www.rabbitmq.com/documentation.html), which is an implementation of AMQP in Erlang. If you want more information about it I let you follow the link. The result is quite good, I managed to have a working POC, adapt to our uses, in a single day. Test plan: 0. Prerequisite: % sudo apt install libanyevent-rabbitmq-perl rabbitmq-server % sudo cpanm Net::RabbitFoot # Quid of the debian package? Then make sure the RabbitMQ server is running: % sudo service rabbitmq-server status and start it if needed with % sudo service rabbitmq-server start Then create, or make sure you have a MARC Modification Template, with template_id=1 (like add new field 123$z=foo) 1. To test this patch set you should first understand how the enqueue/consume process work: % perl new_koha_job.pl # Launch it 1+ times => The job must have been enqueued (no error in the output) % perl koha_worker.pl # Will consume the 1+ tasks you enqueued Do not close this worker and start another one in another console, then enqueue more new jobs. Enjoy :) (yes it should be fun) 2. Watch the history of the jobs Hit admin/background_jobs.pl (logged in as superlibrarian) This is an interface (to improve) to the list of pending/running and finished processes 3. Use this whole stuff in a real Koha world: Play with the "Batch record modification" tool (with biblios) Notice the different in the last (report) step and click on the "View detail of the enqueued job" link Discussions: 1. Please grep TODO and FIXME in the patchset, there are lot to do/fix. 2. Even if we do not go for RabbitMQ I think we could start moving the code from pl to Koha::BackgroundJob::*pm It will be "easy" (nothing is easy, right?) to replace it with something else. 3. With this base we will want to answer different needs: * background jobs * task scheduler * reindex process (?) Going further: 1. Implement Koha::BackgroundJob for the different background jobs 2. Provide a koha-worker script to consume to jobs 3. Rewrite the task scheduler with Koha::BackgroundJob (1993) 4. Provide a configuration file to prioritize or postpone jobs (for instance no batch modification on Wednesday between 9-17) - Or maybe we won't need that (?) 5. What would be your needs? Code is at https://gitlab.com/joubu/Koha/commits/bug_22417 (as the code will evolve in the next days I will not pollute here). Switched to 'In Discussion' to collect thoughts. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 22417] Delegate background jobs execution
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 Fridolin SOMERS changed: What|Removed |Added CC||fridolin.som...@biblibre.co ||m -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
