It does appear that using xdmp:set() on my sequence of elements leads to the apparent commit deadlock.
I reworked my code to iterate over a sequence of task IDs and then in each iteration I fetch the task element using that ID. With that change, my loop succeeds. Using a scheduled job in this case doesn’t really work because I need to load up the client with as many job-submission tasks as there are available threads in the task server to make sure the servers are fully loaded (most jobs complete in a few seconds). So my approach is to divide the total jobs among N tasks in the task server. I do have a scheduled job to recreate these tasks, for example, in the case of a server restart. I could probably adjust my system where the tasks only do a subset of the total jobs and I depend on the scheduled job to create a new batch of job-submission tasks, but that seems unnecessarily complicated. Cheers, E. -- Eliot Kimber http://contrext.com From: <general-boun...@developer.marklogic.com> on behalf of William Sawyer <wilby.saw...@gmail.com> Reply-To: MarkLogic Developer Discussion <general@developer.marklogic.com> Date: Monday, November 27, 2017 at 10:59 AM To: MarkLogic Developer Discussion <general@developer.marklogic.com> Subject: Re: [MarkLogic Dev General] How to Do Equivalent of While true() Loop In ML? You could recursively spawn or setup a schedule task to run every minute or faster if needed. -Will On Mon, Nov 27, 2017 at 9:56 AM, Eliot Kimber <ekim...@contrext.com> wrote: I have a client-server system where the client is spawning 100s of 1000s of jobs on the client. The client polls the servers to see when each server’s task queue is ready for more jobs. This all works fine. Logically this polling is a while-true() loop that will continue until either all the servers are offline or all the tasks to be submitted are consumed. In a procedural language this is trivial, but in XQuery 2 I’m not finding a way to do it that works. In XQuery 3 I could use the new iterate operator but that doesn’t seem to be available in MarkLogic 9. My first attempt was to use a recursive process, relying on tail recursion optimization to avoid blowing the stack buffer. That worked logically but I still ran into out-of-memory on the server at some point (around 200K jobs submitted) and it seems likely that it was runaway recursion doing it. So I tried using a simple loop with xdmp:set() to iterate over the tasks and use an exception to break out when all the tasks are done: try { for $i in 1 to 1000000 (: i.e., loop forever :) if (empty($tasks)) then error() else submit-task(head($tasks)) xdmp:set($tasks, tail($tasks)) } catch ($e) { (: We’re done. ( } Is there a better way to do this kind of looping forever? I’m also having a very strange behavior where in my new looping code I’m getting what I think must be a pending commit deadlock that I didn’t get in my recursive version of the code. I can trace the code to the xdmp:eval() that would commit an update to the task and that code never returns. Each task is a document that I update to reflect the details of the task’s status (start and end times, current processing status, etc.). Those updates are all done either in separately-run modules or via xdmp:eval(), so as far as I can tell there shouldn’t be any issues with uncommitted updates. I didn’t change anything in the logic that updates the task documents, only the loop that iterates over the tasks. Could it be that the use of xdmp:set() to modify the $tasks variable (a sequence of <task> elements) would be causing some kind of commit lock? Thanks, Eliot -- Eliot Kimber http://contrext.com _______________________________________________ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
