Thanks Brad and Danny ! Both the scenarios are working fine . The only part uncleared is the task server queue limitation. Can you please provide some brief impacts of setting high task server queue size. I even tested the scenario with 1M queue size and its working fine.
Thanks a lot again ! Abhishek Srivastav Systems Engineer Tata Consultancy Services Cell:- +91-9883389968 Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Outsourcing ____________________________________________ From: Danny Sokolsky <[email protected]> To: General MarkLogic Developer Discussion <[email protected]> Date: 05/15/2011 04:26 AM Subject: Re: [MarkLogic Dev General] Batch Update Sent by: [email protected] Brad's idea of using corb is probably a good one here. But if you wanted to do the spawn approach, it should work. You might have to raise the task queue size (in the task server configuration) and/or put some number (100 or 1000 for example) of items in each spawned task. -Danny ________________________________________ From: [email protected] [[email protected]] On Behalf Of Rix, Brad [[email protected]] Sent: Saturday, May 14, 2011 12:35 PM To: General MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Batch Update You might look into using Corb (http://developer.marklogic.com/code/corb) This is java based tool that handles threading so that you can control how many to update at one time. You point to an XQuery module for a list of URIs, then another XQuery module that is run on each URI. In your case, your URI list might be generated to be just the list of numbers returned. Then the xquery module for each one would run the insert.xqy with the number as input to each call. While running corb, your task queue will never grow because the Java Corb job manages its own queue. [cid:[email protected]] Brad Rix Senior Developer +1 (303) 542-2172 | Office +1 (303) 915-2771 | Mobile [email protected] http://www.flatironssolutions.com IM: AIM: BradRix MSN: [email protected] From: [email protected] [ mailto:[email protected]] On Behalf Of Abhishek53 S Sent: Saturday, May 14, 2011 1:16 PM To: [email protected] Subject: [MarkLogic Dev General] Batch Update Hi All I need to update millions of records in my marklogic repository. xdmp:spawn is failed to work as at certain point of time the task count in the queue is exceeding 1Lac. The sample query is as follows for $x in (1 to 10000000) return xdmp:spawn("insert.xqy",(xs:QName("filename"),$x)) Thanks in advance! Abhishek Srivastav Systems Engineer Tata Consultancy Services Cell:- +91-9883389968 Mailto: [email protected]<mailto:[email protected]> Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Outsourcing ____________________________________________ =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
_______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
