John, Adding a collection to a document should cost the price of updating a document. You might try using the URI lexicon to get the list of URIs you want to update, and then iterate through that loop to update batches of documents (in batches of say 100 to 1000, depending) by adding the collections. My guess is that your function is locking many more documents than needed. Also, try running it through the profiler to see where it is spending time.
-Danny -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Geert Josten Sent: Thursday, May 31, 2012 7:28 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Collection operations performance Hi John, Not sure where you read that creating a collection would be resource 'intensive'. Working with collections should actually be pretty fast. I think there is something else in your code that is slowing things down. I'm suspecting the xdmp:value could be the cause of this. Can you elaborate on what the getValueQueryStringForISBN is doing? I don't think you would need the xdmp:value. Kind regards, Geert > -----Oorspronkelijk bericht----- > Van: [email protected] [mailto:general- > [email protected]] Namens Ollier, John > Verzonden: donderdag 31 mei 2012 16:14 > Aan: [email protected] > Onderwerp: [MarkLogic Dev General] Collection operations performance > > I have been trying to create/add to a collection of documents with a query > like this: > > define function addTitlesToCollection($collectionUri, $isbnList) { > let $isbn-query-string := > fn:concat('cts:search(fn:collection("my-main-collection"),cts:or-query((', > getValueQueryStringForISBN($isbnList), ')),"unfiltered")') > let $search-documents := xdmp:value($isbn-query-string) > return <result>{ > for $book in $search-documents > let $book-uri := fn:document-uri($book) > let $result := > xdmp:document-add-collections($book-uri,$collectionUri) > return if (fn:empty($result)) > then <success>Successfully adding {$book-uri} to collection > {$collectionUri}</success> > else <error>Error adding {$book-uri} to collection > {$collectionUri}</error> > } > </result> > } > > > It's extremely slow - adding 100 documents to the collection takes 6 minutes > and I'd like to be able to add 1000. > > I appreciate that the Marklogic documentation warns you that creating a > collection is resource intensive. But is there any way of making it faster? > I'm looking at handling the creation process asynchronously, but I'm > concerned the fact that it's so slow means that the performance of Marklogic > will be affected for other applications that use it. What the best way to > check this? > > Thanks in advance. > > John > > > > > ***************************************************************** > *************** > DISCLAIMER: This e-mail is confidential and should not be used by anyone who is > not the original intended recipient. If you have received this e-mail in error > please inform the sender and delete it from your mailbox or any other storage > mechanism. Neither Macmillan Publishers Limited nor any of its agents accept > liability for any statements made which are clearly the sender's own and not > expressly made on behalf of Macmillan Publishers Limited or one of its agents. > Please note that neither Macmillan Publishers Limited nor any of its agents > accept any responsibility for viruses that may be contained in this e-mail or > its attachments and it is your responsibility to scan the e-mail and > attachments (if any). No contracts may be concluded on behalf of Macmillan > Publishers Limited or its agents by means of e-mail communication. Macmillan > Publishers Limited Registered in England and Wales with registered number > 785998 > Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS > ***************************************************************** > *************** > > _______________________________________________ > General mailing list > [email protected] > http://community.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://community.marklogic.com/mailman/listinfo/general
