Thanks everyone for your replies. My statement about adding to collections being expensive was based on this section in MarkLogic Search Developer¹s GuidePage 181 (5October, 2011)
<<<<< quote >>>>> 16.6.2 Adding/Removing Existing Documents To/From Collections A second trade-off with MarkLogic's implementation of collections is that adding or removing documents from collections once those documents are already in the database can be relatively resource-intensive. Changing the collections to which a document belongs requires rewriting every fragment of the document. For large documents, this can be demanding on both CPU and I/O resources. If collection membership is highly dynamic in your application, a better approach may be to use elements within the document itself to characterize membership. <<<<<<>>>>>> I tried running my function in DQ's profiler and I'd need to do some more research to understand the results. However, I have experimented running my function as I add each line and I can see that when I find 20 documents using the ISBN, it takes 0.005 sec, but when I add the line to add each document to the collection, it goes up to 23 sec. John Message: 4 Date: Thu, 31 May 2012 08:43:22 -0700 From: Danny Sokolsky <[email protected]> Subject: Re: [MarkLogic Dev General] Collection operations performance To: MarkLogic Developer Discussion <[email protected]> Message-ID: <c9924d15b04672479b089f7d55ffc13222639e5...@exchg-be.marklogic.com> Content-Type: text/plain; charset="us-ascii" John, Adding a collection to a document should cost the price of updating a document. You might try using the URI lexicon to get the list of URIs you want to update, and then iterate through that loop to update batches of documents (in batches of say 100 to 1000, depending) by adding the collections. My guess is that your function is locking many more documents than needed. Also, try running it through the profiler to see where it is spending time. -Danny -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Geert Josten Sent: Thursday, May 31, 2012 7:28 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Collection operations performance Hi John, Not sure where you read that creating a collection would be resource 'intensive'. Working with collections should actually be pretty fast. I think there is something else in your code that is slowing things down. I'm suspecting the xdmp:value could be the cause of this. Can you elaborate on what the getValueQueryStringForISBN is doing? I don't think you would need the xdmp:value. Kind regards, Geert On 31 May 2012, at 07:14 , Ollier, John wrote: > I have been trying to create/add to a collection of documents with a query > like this: > > define function addTitlesToCollection($collectionUri, $isbnList) { > let $isbn-query-string := > fn:concat('cts:search(fn:collection("my-main-collection"),cts:or-query((', > getValueQueryStringForISBN($isbnList), ')),"unfiltered")') > let $search-documents := xdmp:value($isbn-query-string) > return <result>{ > for $book in $search-documents > let $book-uri := fn:document-uri($book) > let $result := > xdmp:document-add-collections($book-uri,$collectionUri) > return if (fn:empty($result)) > then <success>Successfully adding {$book-uri} to collection > {$collectionUri}</success> > else <error>Error adding {$book-uri} to collection > {$collectionUri}</error> > } > </result> > } > > > It's extremely slow - adding 100 documents to the collection takes 6 minutes > and I'd like to be able to add 1000. > > I appreciate that the Marklogic documentation warns you that creating a > collection is resource intensive. But is there any way of making it faster? > I'm looking at handling the creation process asynchronously, but I'm > concerned the fact that it's so slow means that the performance of Marklogic > will be affected for other applications that use it. What the best way to > check this? > > Thanks in advance. ******************************************************************************** DISCLAIMER: This e-mail is confidential and should not be used by anyone who is not the original intended recipient. If you have received this e-mail in error please inform the sender and delete it from your mailbox or any other storage mechanism. Neither Macmillan Publishers Limited nor any of its agents accept liability for any statements made which are clearly the sender's own and not expressly made on behalf of Macmillan Publishers Limited or one of its agents. Please note that neither Macmillan Publishers Limited nor any of its agents accept any responsibility for viruses that may be contained in this e-mail or its attachments and it is your responsibility to scan the e-mail and attachments (if any). No contracts may be concluded on behalf of Macmillan Publishers Limited or its agents by means of e-mail communication. Macmillan Publishers Limited Registered in England and Wales with registered number 785998 Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS ******************************************************************************** _______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
