The cause of the slowness is physics: you're asking the cluster to do quite a
bit of work. The fix is to find a clever way to accomplish the same goal,
whatever that goal may be. What's the larger goal that you're trying to
accomplish? Got a user story?
The qm:host data shows that you're making a lot of round-trips inside the
cluster, each of which adds network latency. Worse yet, each round-trip could
be driving disk I/O on the other side. So that's a good target to reduce.
What's causing all those round-trips? They could be caused by just about
anything. Profiling the query would show exactly what's happening, and the rest
of the query-meters output would help too. However in this particular case the
sample query shows us that it's probably doing thousands of document fetches.
The sample code basically fetches every document in the directory, however
large it might be. Every document fetch drives a round-trip, and probably disk
I/O on the other side.
We could probably make things a little better by doing the pagination more
effectively. You want that to be in the same function and expression with your
cts:search, like so:
declare function my:search(
$q as cts:query,
$start as xs:integer,
$page-size as xs:integer)
as node()*
{
(: Do the math outside the predicate. :)
let $stop := $start + $page-size
return cts:search(doc(), $query, 'unfiltered')[$start to $stop]
};
Now your original sample becomes something like:
my:search(cts:directory-query($baseUri, 'infinity'), 1, $count)
Or you could get the same results using search:resolve-nodes instead. Its
function signature is similar.
search:resolve-nodes(
document { cts:directory-query($baseUri, 'infinity') }/*,
(), 1, $count)
However it still takes N round-trips to fetch N docs, minus any that are
already in the evaluating host's tree cache. So you're still driving up to
$count round-trips, or one per document in the directory if that's smaller that
$count.
The only way to fix that is to back up and ask broader questions. What's the
goal of fetching $count documents from a directory? What do you plan to do with
those documents? What's the user story? What task is the user trying to
accomplish?
Once we have the problem framed correctly, probably we'll see a more efficient
way to accomplish it.
-- Mike
> On 17 Feb 2015, at 09:01 , Tim Sweetman <[email protected]> wrote:
>
> Hi. Hoping one of you can point me in the right direction here, or at
> least check my conclusions.
>
> We have this query which tends to run about 10x faster on our development
> environments ("cluster" with single MarkLogic server) than on our
> production environment (multiple-server cluster). Trying to track down the
> cause of the slowness using query-meters(), I'm seeing output like:
>
> <qm:host>
> <qm:host-name>HOST003</qm:host-name>
> <qm:round-trip-count>2065</qm:round-trip-count>
> <qm:round-trip-time>PT0.732573S</qm:round-trip-time>
> </qm:host>
> Š
> (2x more hosts)
>
> (On a single-server setup, the entire qm:host output is missing,
> presumably because no inter-server traffic is required to answer the
> query).
>
> I'm not entirely sure how to interpret this output particularly, I'm not
> sure what round-trip-time is measuring.
>
> Some possible approaches we thought of:
> * Ensuring that we're using indexes, "unfiltered" in searches whenever
> possible. (It looks like high round-trip-count values correspond to such
> things as numerous doc() calls, large filtered searches, etc)
> * Is there some way to hint to MarkLogic about spanning fewer servers
> during query execution?
> * Anything else I should look at?
>
> A simplified query I wrote to replicate the problem (needs, say,
> 10000-100000 documents in MarkLogic to work) is below.
>
> Thanks,
>
> Tim S
> (Software Engineer, BBC)
>
> Example script:
> xquery version "1.0-ml";
>
> (: This runs in a fraction-of-a-second on data sets we're using for this
> project. YMMV. :)
> declare function local:large-unfiltered-query($baseUri) {
> cts:search(xdmp:directory-properties($baseUri, "infinity"),
> cts:and-query(()), "unfiltered")
> };
>
> (: High count means longer execution time, and high round-trip-count :)
> let $count := 10000
>
> let $ignore :=
> let $bar := local:large-unfiltered-query("/")
> for $doc in $bar[0 to $count]
> let $retrieved := doc(document-uri($doc))
> return ""
> return xdmp:query-meters()
>
> _______________________________________________
> General mailing list
> [email protected]
> http://developer.marklogic.com/mailman/listinfo/general
>
_______________________________________________
General mailing list
[email protected]
http://developer.marklogic.com/mailman/listinfo/general
