You could recursively spawn or setup a schedule task to run every minute or faster if needed.
-Will On Mon, Nov 27, 2017 at 9:56 AM, Eliot Kimber <[email protected]> wrote: > I have a client-server system where the client is spawning 100s of 1000s > of jobs on the client. The client polls the servers to see when each > server’s task queue is ready for more jobs. This all works fine. > > Logically this polling is a while-true() loop that will continue until > either all the servers are offline or all the tasks to be submitted are > consumed. > > In a procedural language this is trivial, but in XQuery 2 I’m not finding > a way to do it that works. In XQuery 3 I could use the new iterate operator > but that doesn’t seem to be available in MarkLogic 9. > > My first attempt was to use a recursive process, relying on tail recursion > optimization to avoid blowing the stack buffer. That worked logically but I > still ran into out-of-memory on the server at some point (around 200K jobs > submitted) and it seems likely that it was runaway recursion doing it. > > So I tried using a simple loop with xdmp:set() to iterate over the tasks > and use an exception to break out when all the tasks are done: > > try { > for $i in 1 to 1000000 (: i.e., loop forever :) > if (empty($tasks)) > then error() > else submit-task(head($tasks)) > xdmp:set($tasks, tail($tasks)) > } catch ($e) { > (: We’re done. ( > } > > Is there a better way to do this kind of looping forever? > > I’m also having a very strange behavior where in my new looping code I’m > getting what I think must be a pending commit deadlock that I didn’t get in > my recursive version of the code. I can trace the code to the xdmp:eval() > that would commit an update to the task and that code never returns. > > Each task is a document that I update to reflect the details of the task’s > status (start and end times, current processing status, etc.). Those > updates are all done either in separately-run modules or via xdmp:eval(), > so as far as I can tell there shouldn’t be any issues with uncommitted > updates. I didn’t change anything in the logic that updates the task > documents, only the loop that iterates over the tasks. > > Could it be that the use of xdmp:set() to modify the $tasks variable (a > sequence of <task> elements) would be causing some kind of commit lock? > > Thanks, > > Eliot > > -- > Eliot Kimber > http://contrext.com > > > > > _______________________________________________ > General mailing list > [email protected] > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general >
_______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
