Hi Allen,
You could use xdmp:directory-delete('/documents/7615990080550346622/'), but
what is killing you here is the CPF triggers. Each document delete is
triggering CPF events, which consumes a lot of unnecessary time and resources.
You could try to short-cut delete events, or should consider temporarily
disabling CPF to do the deletion. Another option is if you would have separate
CPF domains for each collection, you could drop that to exclude it from CPF
before you start deleting..
Kind regards,
Geert
Van: [email protected]
[mailto:[email protected]] Namens Tim Meagher
Verzonden: dinsdag 7 juni 2011 5:19
Aan: 'General MarkLogic Developer Discussion'
Onderwerp: Re: [MarkLogic Dev General] Deleting large number of documents from
large database
I use cts:uris to get a list of the uris that I want to delete and then I spawn
an xquery to delete each document. You have to be careful that you don't spawn
more deletion tasks than can be accommodated on the task server queue.
Tim Meagher
From: [email protected]
[mailto:[email protected]] On Behalf Of
[email protected]
Sent: Monday, June 06, 2011 10:07 PM
To: Marklogic Developer Mailing List
Subject: [MarkLogic Dev General] Deleting large number of documents from large
database
I need to delete all of the documents in a collection in Marklogic. The
database in which we have the documents stored has over 4.4 million documents.
The collection itself has approximately 1.7 million documents. We are
currently running 4.0-4 of MarkLogic. We will be upgrading in the near future
to 4.2, but that does not help us with this problem.
I have attempted to use the code below to perform the delete, but I am only
able to delete 5 documents per second using this method.
(: Delete documents in a collection :)
xquery version "1.0-ml";
let $assert :=
if (xdmp:request-timestamp()) then ()
else
error((), 'NOTIMESTAMP', text { 'outer query is not read-only' })
let $path := '/documents/7615990080550346622/'
let $map := map:map()
let $list-uris :=
for $i in xdmp:directory($path, 'infinity')[1 to 10000]
return map:put($map, xdmp:node-uri($i), true())
let $do := xdmp:eval('
declare variable $URIS as map:map external;
xdmp:document-delete(map:keys($URIS))
',
(xs:QName('URIS'), $map),
<options xmlns="xdmp:eval">
<isolation>different-transaction</isolation>
<prevent-deadlocks>true</prevent-deadlocks>
</options>
)
return count(map:keys($map))
I read another post on this mailing list that indicated that if you set
directory creation to "manual" and did not have any triggers enabled in the
database, that doing a directory delete would be extremely fast. However, we
DO have CPF enabled (and cannot disable it) and cannot set directory creation
to manual since we depend on the automatic directory creation in our
application.
Any suggestions as to how we can get better throughput for deleting all of the
documents in a collection OR deleting a large number of documents from a large
database? Clearing a collection as quickly as possible while retaining the
collection itself is the goal at this time.
Thanks,
Allen Shatzer
_______________________________________________
General mailing list
[email protected]
http://developer.marklogic.com/mailman/listinfo/general
