HI Justin,
Thanks for your quick reply. However, I used
both declareUpdate({explicitCommit: true}); and declareUpdate(); Still
same issue.
I am just reading from one collection and updating into another document in
a different collection. I managed with insertDocument, but this over
writes the docs which I don't want. I just want to update the attributes (
it is like UPDATE SET field1:=value, field2=value in loop in oracle world)
Note: I used lock/unlock options, but getting no lock on that target
record. I used isolation level as part of eval. Couldn't get through it.
XDMP-CONFLICTINGUPDATES: xdmp.eval("//{explicitCommit:
true}\ndeclareUpdate(); \nvar json = \"/Mark...", {}, <options
xmlns="xdmp:eval"><database>67298589469423423423</database><root>/</root><isolation...</options>)
--Conflicting updates xdmp.nodeReplace(cts.doc("
Any thoughts?
On Thu, Sep 29, 2016 at 7:55 AM, <[email protected]>
wrote:
> Send General mailing list submissions to
> [email protected]
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://developer.marklogic.com/mailman/listinfo/general
> or, via email, send a message with subject or body 'help' to
> [email protected]
>
> You can reach the person managing the list at
> [email protected]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of General digest..."
>
>
> Today's Topics:
>
> 1. Re: XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace
> (Justin Makeig)
> 2. Re: mlcp Transaction Errors - SVC-EXTIME and XDMP-NOTXN #CGO#
> (Jain, Abhishek)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 28 Sep 2016 21:02:26 +0000
> From: Justin Makeig <[email protected]>
> Subject: Re: [MarkLogic Dev General] XDMP-CONFLICTINGUPDATES: while
> using xdmp.nodeReplace
> To: MarkLogic Developer Discussion <[email protected]>
> Message-ID: <[email protected]>
> Content-Type: text/plain; charset="us-ascii"
>
> The general pattern for doing updates on persisted JSON documents in
> JavaScript is to
> 1) get the document
> 2) turn it into an object
> 3) update the object
> 4) replace the document node with the updated object
>
> Here's an example that updates the `balance` property of every document in
> the `accounts` collection:
>
> declareUpdate();
> for(var doc of fn.collection('accounts')) {
> var account = doc.toObject();
> account.balance = account.balance * 1.05;
> xdmp.nodeReplace(doc, account);
> }
>
> Note that you also need a declareUpdate(). The transaction is committed
> (or rolledback) automatically, so you don't need the xdmp.commit().
>
> Justin
>
> --
> Justin Makeig
> Director, Product Management
> MarkLogic
> [email protected]
>
> > On Sep 28, 2016, at 1:46 PM, Shiv Shankar <[email protected]>
> wrote:
> >
> > Hi Rob,
> > Thanks for quick reply,
> >
> > Here is the sample scenario.
> >
> > I would be building random data and put in to newObjData in a for loop.
> >
> > 01.json = {"firstName": "abc", "lastName": "xyz"}
> >
> > newObjectData.firstName= "Donald";
> >
> > for Loop
> > {
> > xdmp.nodeReplace(cts.doc("/test/01.json"),newObjectData);
> > xdmp.commit();
> > }
> >
> >
> >
> >
> >
> > On Wed, Sep 28, 2016 at 4:37 PM, Rob Szkutak <[email protected]>
> wrote:
> > Hi,
> >
> > Do you have a reproducible case of this I could look at?
> >
> > Best,
> > Rob
> >
> > Rob Szkutak
> > Senior Consultant
> > MarkLogic Corporation
> > [email protected]
> > www.marklogic.com
> >
> > From: Shiv Shankar [[email protected]]
> > Sent: Wednesday, September 28, 2016 3:35 PM
> > To: [email protected]; Rob Szkutak
> > Subject: XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace
> >
> > Hi Rob,
> >
> > I am getting XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace to
> update JSON attributes in FOR loop, instead of over-writing entire
> document. Any advice.
> >
> > BTW, I used xdmp.eval, but it is not identifying the java script
> variables.
> >
> > Thanks
> > ShivShankar.
> >
> > _______________________________________________
> > General mailing list
> > [email protected]
> > Manage your subscription at:
> > http://developer.marklogic.com/mailman/listinfo/general
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 29 Sep 2016 11:54:54 +0000
> From: "Jain, Abhishek" <[email protected]>
> Subject: Re: [MarkLogic Dev General] mlcp Transaction Errors -
> SVC-EXTIME and XDMP-NOTXN #CGO#
> To: MarkLogic Developer Discussion <[email protected]>
> Message-ID:
> <5A37843F322E824AA0D8359ED88CD01F276FF29D@IN-HYD-DAGND2.
> corp.capgemini.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> Great , Expected !! As it was pretty optimized MLCP Query.
> BTW, addressing the second part of your question about logging
> errors/failed files ?
> I?m also working on similar stuff, this can be done in three ways ?
>
>
> 1. Modifying MLCP library to write all the failed IDs and
> corresponding files in a separate log file
>
> However that requires additional work(java program or x-query ) program to
> create new input files of failed record.(Tedious)
>
> 2. Second way, we need to write a pre-commit trigger, and attach to
> database. In pre-commit action module we can
>
> Write custom logs.
>
> 3. Similar module can also be written for MLCP Transform.
>
>
>
> All three approaches have limitations , which one to choose depends on
> what exactly we want to log.
>
> In my case it pre-commit solution does well.
>
> I wonder MLCP doesn?t provide RE-TRY option.
>
> Thanks and Regards,
> [Email_CBE.gif]Abhishek Jain
> Associate Consultant
>
> From: [email protected] [mailto:general-bounces@
> developer.marklogic.com] On Behalf Of Stuart Myles
> Sent: Friday, September 23, 2016 8:45 PM
> To: MarkLogic Developer Discussion
> Subject: Re: [MarkLogic Dev General] mlcp Transaction Errors - SVC-EXTIME
> and XDMP-NOTXN #CGO#
>
> Thanks! This helped me prevent the errors from occurring and - as a bonus
> - significantly sped up my ingestion.
>
> I couldn't use exactly the mlcp command line you suggested, since - in the
> version of mlcp I'm using - -input_file_type xml isn't allowed, I had to
> use -input_file_type documents instead. Also, my input files don't need to
> be split. However, bumping up the threads used (to 30 in my case) made the
> transaction / timeout complaints go away. And now I'm ingesting 100,000
> documents in 12 minutes, rather than one hour. Much better!
>
> Regards,
>
> Stuart
>
>
>
> On Fri, Sep 23, 2016 at 3:34 AM, Jain, Abhishek <
> [email protected]<mailto:[email protected]>>
> wrote:
> Hi Stuart,
>
> MLCP comes with various options, and can be used in various combinations
> depending on the file size, memory available and
> Other number of nodes, forest etc.
>
> If you want to try a quick solution you can try this mlcp command :
> mlcp import -host yourhost -port 8000 -username userName -password
> PASSWORD -input_file_type xml -input_file_path TempData -thread_count
> -thread_count_per_split 3 -batch_size 200 -transaction_size 20
> -max_split_size 33554432 -split_input true
> change username, input file type etc accordingly.
> It?s always good to use splits and threads when working with huge dataset.
> Some performance matrix you can consider while using above mlcp :
>
> 1. In app server settings you can check if connection time out is
> set to 0.
>
> 2. Default spilt size is 32MB, if you can change -max_split_size
> 33554432 ( it take in bytes, if your file is bigger )
>
> 3. Make sure split and thread ratio remains 1:2 or 1:3 for example
> If your document size is 10 MB, and your split size is 1000,000 (1 MB)
> then 10/1 = 10 splits
> Then you should create 20 or 30 thread for best CPU utilization.
>
> 4. The above mlcp does well with 150 Million rows, should work for
> you as well.
>
> 5. I assume you have a nice good RAM > 4GB alteast.
>
> Thanks and Regards,
> [Email_CBE.gif]Abhishek Jain
> Associate Consultant
> Capgemini India | Hyderabad
>
> From: [email protected]<mailto:general-
> [email protected]> [mailto:general-bounces@
> developer.marklogic.com<mailto:[email protected]>]
> On Behalf Of Stuart Myles
> Sent: Thursday, September 22, 2016 11:52 PM
> To: MarkLogic Developer Discussion
> Subject: [MarkLogic Dev General] mlcp Transaction Errors - SVC-EXTIME and
> XDMP-NOTXN
>
> When I'm loading directories of slightly fewer than 100,000 XML files into
> a large MarkLogic instance, I often get timeout and transaction errors. If
> I re-run the same directory of files which got those errors, I typically
> don't get any errors.
>
> So, I have a few questions:
>
> * Can I get prevent the errors from happening in the first place - e.g. by
> tuning MarkLogic parameters or altering my use of mlcp?
> * If I do get errors, what is the best way to get a report on the files
> which failed, so I can retry just those ones? Is the best option for me to
> write some code to pick out the errors from the log file? And, if so, am I
> guaranteed to get all of the files reported?
>
> Some Details
>
> The command line template is
>
> mlcp.sh import -username {1} -password {2} -host localhost -port {4}
> -input_file_path {5} -output_uri_replace \"{6},'{7}'\"
>
> Sometimes, the imports run just fine. However, often I get a large number
> of SVC-EXTIME errors followed by a XDMP-NOTXN error. For example:
>
> 16/09/22 17:54:03 ERROR mapreduce.ContentWriter: SVC-EXTIME: Time limit
> exceeded
> 16/09/22 17:54:03 WARN mapreduce.ContentWriter: Failed document
> 029ccd8ac3323658277ca28fead7a73d.0.xml in file:/mnt/ingestion/
> MarkLogicIngestion/smyles/todo/2014_0005.done/
> 029ccd8ac3323658277ca28fead7a73d.0.xml
> 16/09/22 17:54:03 ERROR mapreduce.ContentWriter: SVC-EXTIME: Time limit
> exceeded
> 16/09/22 17:54:03 WARN mapreduce.ContentWriter: Failed document
> 02eb4562784255e249c4ec3ed472f9aa.1.xml in file:/mnt/ingestion/
> MarkLogicIngestion/smyles/todo/2014_0005.done/
> 02eb4562784255e249c4ec3ed472f9aa.1.xml
> 16/09/22 17:54:04 INFO contentpump.LocalJobRunner: completed 33%
> 16/09/22 17:54:21 ERROR mapreduce.ContentWriter: XDMP-NOTXN: No
> transaction with identifier 9076269665213828952
>
> So far, I'm just rerunning the entire directory again. Most of the time,
> it ingests fine on the second attempt. However, I have thousands of these
> directories to process. So, I would prefer to avoid getting the errors in
> the first place. Failing that, I would like to capture the errors and just
> retry the files which failed.
>
> Any help much appreciated.
>
> Regards,
>
> Stuart
>
>
>
> This message contains information that may be privileged or confidential
> and is the property of the Capgemini Group. It is intended only for the
> person to whom it is addressed. If you are not the intended recipient, you
> are not authorized to read, print, retain, copy, disseminate, distribute,
> or use this message or any part thereof. If you receive this message in
> error, please notify the sender immediately and delete all copies of this
> message.
>
> _______________________________________________
> General mailing list
> [email protected]<mailto:[email protected]>
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://developer.marklogic.com/pipermail/general/
> attachments/20160929/e5f311a1/attachment.html
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image001.gif
> Type: image/gif
> Size: 1616 bytes
> Desc: image001.gif
> Url : http://developer.marklogic.com/pipermail/general/
> attachments/20160929/e5f311a1/attachment.gif
>
> ------------------------------
>
> _______________________________________________
> General mailing list
> [email protected]
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
>
> End of General Digest, Vol 147, Issue 51
> ****************************************
>
_______________________________________________
General mailing list
[email protected]
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general
