I can see how this form of recursive function using xdmp:set() is better than
for 1 to 1,000,000.
I was trying to find a pure XQuery solution, that is, one that didn’t rely on
xdmp:set(), so my recursion was:
declare function local:handle-task($task, $tasks) {
if (empty($task))
then ()
else
let $do-submit := local:submit-job($task)
return local:handle-task(head($tasks), tail($tasks))
}
(I could of course just pass tasks but I think the function signature is
clearer if the thing that is acted on is passed as a separate parameter.)
But beyond that I have another loop or recursion that polls the remote servers
until a server becomes available. That recursion I think was where I was
getting out of memory even though tail recursion optimization should have
prevented it.
So in my mind there’s a still a question of whether or not a pure XQuery
solution is possible with ML 9 or do I have to use xdmp:set()?
Of course I have to do whatever will solve the problem but I like to avoid
proprietary extensions whenever possible (otherwise, what’s the point of using
standards).
Cheers,
Eliot
--
Eliot Kimber
http://contrext.com
On 11/28/17, 10:39 AM, "general-boun...@developer.marklogic.com on behalf of
John Snelson" <general-boun...@developer.marklogic.com on behalf of
john.snel...@marklogic.com> wrote:
You should use recursive functions for this kind of thing:
declare function local:while($test, $body) {
if($test()) then ($body(), local:while($test,$body)) else ()
};
let $tasks := ...
return
local:while(
function() { exists($tasks) },
function() {
submit-task(head($tasks)),
xdmp:set($tasks,tail($tasks))
}
)
MarkLogic's tail call optimization will mean that the local:while()
function will use a constant amount of stack space.
However in your specific example you really just want to execute a
function on each member of a sequence. In that specific case you can use
fn:map:
fn:map(submit-task#1,$tasks)
John
On 27/11/17 16:56, Eliot Kimber wrote:
> I have a client-server system where the client is spawning 100s of 1000s
of jobs on the client. The client polls the servers to see when each server’s
task queue is ready for more jobs. This all works fine.
>
> Logically this polling is a while-true() loop that will continue until
either all the servers are offline or all the tasks to be submitted are
consumed.
>
> In a procedural language this is trivial, but in XQuery 2 I’m not finding
a way to do it that works. In XQuery 3 I could use the new iterate operator but
that doesn’t seem to be available in MarkLogic 9.
>
> My first attempt was to use a recursive process, relying on tail
recursion optimization to avoid blowing the stack buffer. That worked logically
but I still ran into out-of-memory on the server at some point (around 200K
jobs submitted) and it seems likely that it was runaway recursion doing it.
>
> So I tried using a simple loop with xdmp:set() to iterate over the tasks
and use an exception to break out when all the tasks are done:
>
> try {
> for $i in 1 to 1000000 (: i.e., loop forever :)
> if (empty($tasks))
> then error()
> else submit-task(head($tasks))
> xdmp:set($tasks, tail($tasks))
> } catch ($e) {
> (: We’re done. (
> }
>
> Is there a better way to do this kind of looping forever?
>
> I’m also having a very strange behavior where in my new looping code I’m
getting what I think must be a pending commit deadlock that I didn’t get in my
recursive version of the code. I can trace the code to the xdmp:eval() that
would commit an update to the task and that code never returns.
>
> Each task is a document that I update to reflect the details of the
task’s status (start and end times, current processing status, etc.). Those
updates are all done either in separately-run modules or via xdmp:eval(), so as
far as I can tell there shouldn’t be any issues with uncommitted updates. I
didn’t change anything in the logic that updates the task documents, only the
loop that iterates over the tasks.
>
> Could it be that the use of xdmp:set() to modify the $tasks variable (a
sequence of <task> elements) would be causing some kind of commit lock?
>
> Thanks,
>
> Eliot
>
> --
> Eliot Kimber
> http://contrext.com
>
>
>
>
> _______________________________________________
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
--
John Snelson, Principal Engineer http://twitter.com/jpcs
MarkLogic Corporation http://www.marklogic.com
_______________________________________________
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general
