Allen, Certainly moving to something URIs based will help avoid the initial transaction loading the full documents. Batching the deletes may also help (100 or 1000 per transaction, rather than 1 per transaction in your eval()).
The CPF processing may be the culprit. Does CPF run on this domain, and if so is it working hard to figure out whether to run or not when these documents are deleted, or doing actual work? Monitoring your task server for excessive activity can shed light on the task server load. Also logging start and end-times for your processes if you have that in place. Yours, Damon From: [email protected] [mailto:[email protected]] On Behalf Of [email protected] Sent: Monday, June 06, 2011 10:07 PM To: Marklogic Developer Mailing List Subject: [MarkLogic Dev General] Deleting large number of documents from large database I need to delete all of the documents in a collection in Marklogic. The database in which we have the documents stored has over 4.4 million documents. The collection itself has approximately 1.7 million documents. We are currently running 4.0-4 of MarkLogic. We will be upgrading in the near future to 4.2, but that does not help us with this problem. I have attempted to use the code below to perform the delete, but I am only able to delete 5 documents per second using this method. (: Delete documents in a collection :) xquery version "1.0-ml"; let $assert := if (xdmp:request-timestamp()) then () else error((), 'NOTIMESTAMP', text { 'outer query is not read-only' }) let $path := '/documents/7615990080550346622/' let $map := map:map() let $list-uris := for $i in xdmp:directory($path, 'infinity')[1 to 10000] return map:put($map, xdmp:node-uri($i), true()) let $do := xdmp:eval(' declare variable $URIS as map:map external; xdmp:document-delete(map:keys($URIS)) ', (xs:QName('URIS'), $map), <options xmlns="xdmp:eval"> <isolation>different-transaction</isolation> <prevent-deadlocks>true</prevent-deadlocks> </options> ) return count(map:keys($map)) I read another post on this mailing list that indicated that if you set directory creation to "manual" and did not have any triggers enabled in the database, that doing a directory delete would be extremely fast. However, we DO have CPF enabled (and cannot disable it) and cannot set directory creation to manual since we depend on the automatic directory creation in our application. Any suggestions as to how we can get better throughput for deleting all of the documents in a collection OR deleting a large number of documents from a large database? Clearing a collection as quickly as possible while retaining the collection itself is the goal at this time. Thanks, Allen Shatzer
_______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
