Is it possible to route a task to a specific worker? I looked at using priorities but I don't think that is an option/implemented.
- I have 3 celery servers for a total of 55 workers/threads: hostA/13, hostB/21, and hostC/21 (concurrency is after the /) - I'm using RabbitMQ - I'm ETL'ing ~2TB of MySQL data to Riak. - Two workers/threads can't modify the same data at once, so I have a locking mechanism. - The transformation/merging of data into Riak can take several minutes. If 2 or more tasks are working on the same "key", the other has to wait until the merging is complete. This is very common because I create the tasks sequentially based off key. This often leads to many or all workers waiting/idle (for many minutes ... it some cases, several hours). My logic is that if I could route a "key" to a specific worker/thread, I would eliminate the locking. Thanks for any help. Kind regards, -Justin
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
