I still don't understand what your code is trying to do. Can you please describe in words what you're trying to accomplish? Where does `matViewUri` come from? How does it relate to `{personId: id}`?
Justin > On Oct 3, 2016, at 7:08 AM, Shiv Shankar <shiv.shivshan...@gmail.com> wrote: > > Hi Justin, > Here is the actual code that returns XDMP-CONFLICTINGUPDATES Error:. > > --- > declareUpdate(); > var mainDataResponse = > rawColls.documents() > .where(rawColls.byExample({personId: id})) > .result('iterator'); > > for (var result of mainDataResponse.results) { > > var normalizedViewObj = cts.doc(matViewUri).toObject(); > > var matOriginalDoc = cts.doc(matViewUri); > > var dateNow =new Date().toUTCString(); > > // if (doc.firstname !=null) > normalizedViewObj.firstName = result.document.firstname; > > // if (doc.lastName !=null) > normalizedViewObj.lastName = result.document.lastname; > // if (doc.dateOfBirth !=null) > normalizedViewObj.dateOfBirth = result.document.dob; > > xdmp.nodeReplace(cts.doc(matViewUri),normalizedViewObj), > normalizedViewObj = null; > > } > > --- > > On Thu, Sep 29, 2016 at 3:00 PM, <general-requ...@developer.marklogic.com> > wrote: > Send General mailing list submissions to > general@developer.marklogic.com > > To subscribe or unsubscribe via the World Wide Web, visit > http://developer.marklogic.com/mailman/listinfo/general > or, via email, send a message with subject or body 'help' to > general-requ...@developer.marklogic.com > > You can reach the person managing the list at > general-ow...@developer.marklogic.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of General digest..." > > > Today's Topics: > > 1. Re: General Digest, Vol 147, Issue 51 (Justin Makeig) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 29 Sep 2016 16:32:44 +0000 > From: Justin Makeig <justin.mak...@marklogic.com> > Subject: Re: [MarkLogic Dev General] General Digest, Vol 147, Issue 51 > To: Shiv Shankar <shiv.shivshan...@gmail.com> > Cc: "general@developer.marklogic.com" > <general@developer.marklogic.com> > Message-ID: <22fded82-fc67-4aad-b1f9-2eecb5b6a...@marklogic.com> > Content-Type: text/plain; charset="us-ascii" > > Can you show your actual code? > > Justin > > > -- > Justin Makeig > Director, Product Management > MarkLogic > jmak...@marklogic.com > > > > On Sep 29, 2016, at 7:09 AM, Shiv Shankar <shiv.shivshan...@gmail.com> > > wrote: > > > > HI Justin, > > Thanks for your quick reply. However, I used both > > declareUpdate({explicitCommit: true}); and declareUpdate(); Still same > > issue. > > > > I am just reading from one collection and updating into another document in > > a different collection. I managed with insertDocument, but this over writes > > the docs which I don't want. I just want to update the attributes ( it is > > like UPDATE SET field1:=value, field2=value in loop in oracle world) > > > > Note: I used lock/unlock options, but getting no lock on that target > > record. I used isolation level as part of eval. Couldn't get through it. > > XDMP-CONFLICTINGUPDATES: xdmp.eval("//{explicitCommit: > > true}\ndeclareUpdate(); \nvar json = \"/Mark...", {}, <options > > xmlns="xdmp:eval"><database>67298589469423423423</database><root>/</root><isolation...</options>) > > --Conflicting updates xdmp.nodeReplace(cts.doc(" > > > > > > Any thoughts? > > > > On Thu, Sep 29, 2016 at 7:55 AM, <general-requ...@developer.marklogic.com> > > wrote: > > Send General mailing list submissions to > > general@developer.marklogic.com > > > > To subscribe or unsubscribe via the World Wide Web, visit > > http://developer.marklogic.com/mailman/listinfo/general > > or, via email, send a message with subject or body 'help' to > > general-requ...@developer.marklogic.com > > > > You can reach the person managing the list at > > general-ow...@developer.marklogic.com > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of General digest..." > > > > > > Today's Topics: > > > > 1. Re: XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace > > (Justin Makeig) > > 2. Re: mlcp Transaction Errors - SVC-EXTIME and XDMP-NOTXN #CGO# > > (Jain, Abhishek) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Wed, 28 Sep 2016 21:02:26 +0000 > > From: Justin Makeig <justin.mak...@marklogic.com> > > Subject: Re: [MarkLogic Dev General] XDMP-CONFLICTINGUPDATES: while > > using xdmp.nodeReplace > > To: MarkLogic Developer Discussion <general@developer.marklogic.com> > > Message-ID: <4136b8ca-113a-4373-b6d3-e54d34f15...@marklogic.com> > > Content-Type: text/plain; charset="us-ascii" > > > > The general pattern for doing updates on persisted JSON documents in > > JavaScript is to > > 1) get the document > > 2) turn it into an object > > 3) update the object > > 4) replace the document node with the updated object > > > > Here's an example that updates the `balance` property of every document in > > the `accounts` collection: > > > > declareUpdate(); > > for(var doc of fn.collection('accounts')) { > > var account = doc.toObject(); > > account.balance = account.balance * 1.05; > > xdmp.nodeReplace(doc, account); > > } > > > > Note that you also need a declareUpdate(). The transaction is committed (or > > rolledback) automatically, so you don't need the xdmp.commit(). > > > > Justin > > > > -- > > Justin Makeig > > Director, Product Management > > MarkLogic > > jmak...@marklogic.com > > > > > On Sep 28, 2016, at 1:46 PM, Shiv Shankar <shiv.shivshan...@gmail.com> > > > wrote: > > > > > > Hi Rob, > > > Thanks for quick reply, > > > > > > Here is the sample scenario. > > > > > > I would be building random data and put in to newObjData in a for loop. > > > > > > 01.json = {"firstName": "abc", "lastName": "xyz"} > > > > > > newObjectData.firstName= "Donald"; > > > > > > for Loop > > > { > > > xdmp.nodeReplace(cts.doc("/test/01.json"),newObjectData); > > > xdmp.commit(); > > > } > > > > > > > > > > > > > > > > > > On Wed, Sep 28, 2016 at 4:37 PM, Rob Szkutak <rob.szku...@marklogic.com> > > > wrote: > > > Hi, > > > > > > Do you have a reproducible case of this I could look at? > > > > > > Best, > > > Rob > > > > > > Rob Szkutak > > > Senior Consultant > > > MarkLogic Corporation > > > rob.szku...@marklogic.com > > > www.marklogic.com > > > > > > From: Shiv Shankar [shiv.shivshan...@gmail.com] > > > Sent: Wednesday, September 28, 2016 3:35 PM > > > To: general@developer.marklogic.com; Rob Szkutak > > > Subject: XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace > > > > > > Hi Rob, > > > > > > I am getting XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace to > > > update JSON attributes in FOR loop, instead of over-writing entire > > > document. Any advice. > > > > > > BTW, I used xdmp.eval, but it is not identifying the java script > > > variables. > > > > > > Thanks > > > ShivShankar. > > > > > > _______________________________________________ > > > General mailing list > > > General@developer.marklogic.com > > > Manage your subscription at: > > > http://developer.marklogic.com/mailman/listinfo/general > > > > > > > > > > ------------------------------ > > > > Message: 2 > > Date: Thu, 29 Sep 2016 11:54:54 +0000 > > From: "Jain, Abhishek" <abhishek.b.j...@capgemini.com> > > Subject: Re: [MarkLogic Dev General] mlcp Transaction Errors - > > SVC-EXTIME and XDMP-NOTXN #CGO# > > To: MarkLogic Developer Discussion <general@developer.marklogic.com> > > Message-ID: > > > > <5a37843f322e824aa0d8359ed88cd01f276ff...@in-hyd-dagnd2.corp.capgemini.com> > > > > Content-Type: text/plain; charset="utf-8" > > > > Great , Expected !! As it was pretty optimized MLCP Query. > > BTW, addressing the second part of your question about logging > > errors/failed files ? > > I?m also working on similar stuff, this can be done in three ways ? > > > > > > 1. Modifying MLCP library to write all the failed IDs and > > corresponding files in a separate log file > > > > However that requires additional work(java program or x-query ) program to > > create new input files of failed record.(Tedious) > > > > 2. Second way, we need to write a pre-commit trigger, and attach to > > database. In pre-commit action module we can > > > > Write custom logs. > > > > 3. Similar module can also be written for MLCP Transform. > > > > > > > > All three approaches have limitations , which one to choose depends on what > > exactly we want to log. > > > > In my case it pre-commit solution does well. > > > > I wonder MLCP doesn?t provide RE-TRY option. > > > > Thanks and Regards, > > [Email_CBE.gif]Abhishek Jain > > Associate Consultant > > > > From: general-boun...@developer.marklogic.com > > [mailto:general-boun...@developer.marklogic.com] On Behalf Of Stuart Myles > > Sent: Friday, September 23, 2016 8:45 PM > > To: MarkLogic Developer Discussion > > Subject: Re: [MarkLogic Dev General] mlcp Transaction Errors - SVC-EXTIME > > and XDMP-NOTXN #CGO# > > > > Thanks! This helped me prevent the errors from occurring and - as a bonus - > > significantly sped up my ingestion. > > > > I couldn't use exactly the mlcp command line you suggested, since - in the > > version of mlcp I'm using - -input_file_type xml isn't allowed, I had to > > use -input_file_type documents instead. Also, my input files don't need to > > be split. However, bumping up the threads used (to 30 in my case) made the > > transaction / timeout complaints go away. And now I'm ingesting 100,000 > > documents in 12 minutes, rather than one hour. Much better! > > > > Regards, > > > > Stuart > > > > > > > > On Fri, Sep 23, 2016 at 3:34 AM, Jain, Abhishek > > <abhishek.b.j...@capgemini.com<mailto:abhishek.b.j...@capgemini.com>> wrote: > > Hi Stuart, > > > > MLCP comes with various options, and can be used in various combinations > > depending on the file size, memory available and > > Other number of nodes, forest etc. > > > > If you want to try a quick solution you can try this mlcp command : > > mlcp import -host yourhost -port 8000 -username userName -password PASSWORD > > -input_file_type xml -input_file_path TempData -thread_count > > -thread_count_per_split 3 -batch_size 200 -transaction_size 20 > > -max_split_size 33554432 -split_input true > > change username, input file type etc accordingly. > > It?s always good to use splits and threads when working with huge dataset. > > Some performance matrix you can consider while using above mlcp : > > > > 1. In app server settings you can check if connection time out is set > > to 0. > > > > 2. Default spilt size is 32MB, if you can change -max_split_size > > 33554432 ( it take in bytes, if your file is bigger ) > > > > 3. Make sure split and thread ratio remains 1:2 or 1:3 for example > > If your document size is 10 MB, and your split size is 1000,000 (1 MB) then > > 10/1 = 10 splits > > Then you should create 20 or 30 thread for best CPU utilization. > > > > 4. The above mlcp does well with 150 Million rows, should work for > > you as well. > > > > 5. I assume you have a nice good RAM > 4GB alteast. > > > > Thanks and Regards, > > [Email_CBE.gif]Abhishek Jain > > Associate Consultant > > Capgemini India | Hyderabad > > > > From: > > general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> > > > > [mailto:general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>] > > On Behalf Of Stuart Myles > > Sent: Thursday, September 22, 2016 11:52 PM > > To: MarkLogic Developer Discussion > > Subject: [MarkLogic Dev General] mlcp Transaction Errors - SVC-EXTIME and > > XDMP-NOTXN > > > > When I'm loading directories of slightly fewer than 100,000 XML files into > > a large MarkLogic instance, I often get timeout and transaction errors. If > > I re-run the same directory of files which got those errors, I typically > > don't get any errors. > > > > So, I have a few questions: > > > > * Can I get prevent the errors from happening in the first place - e.g. by > > tuning MarkLogic parameters or altering my use of mlcp? > > * If I do get errors, what is the best way to get a report on the files > > which failed, so I can retry just those ones? Is the best option for me to > > write some code to pick out the errors from the log file? And, if so, am I > > guaranteed to get all of the files reported? > > > > Some Details > > > > The command line template is > > > > mlcp.sh import -username {1} -password {2} -host localhost -port {4} > > -input_file_path {5} -output_uri_replace \"{6},'{7}'\" > > > > Sometimes, the imports run just fine. However, often I get a large number > > of SVC-EXTIME errors followed by a XDMP-NOTXN error. For example: > > > > 16/09/22 17:54:03 ERROR mapreduce.ContentWriter: SVC-EXTIME: Time limit > > exceeded > > 16/09/22 17:54:03 WARN mapreduce.ContentWriter: Failed document > > 029ccd8ac3323658277ca28fead7a73d.0.xml in > > file:/mnt/ingestion/MarkLogicIngestion/smyles/todo/2014_0005.done/029ccd8ac3323658277ca28fead7a73d.0.xml > > 16/09/22 17:54:03 ERROR mapreduce.ContentWriter: SVC-EXTIME: Time limit > > exceeded > > 16/09/22 17:54:03 WARN mapreduce.ContentWriter: Failed document > > 02eb4562784255e249c4ec3ed472f9aa.1.xml in > > file:/mnt/ingestion/MarkLogicIngestion/smyles/todo/2014_0005.done/02eb4562784255e249c4ec3ed472f9aa.1.xml > > 16/09/22 17:54:04 INFO contentpump.LocalJobRunner: completed 33% > > 16/09/22 17:54:21 ERROR mapreduce.ContentWriter: XDMP-NOTXN: No transaction > > with identifier 9076269665213828952 > > > > So far, I'm just rerunning the entire directory again. Most of the time, it > > ingests fine on the second attempt. However, I have thousands of these > > directories to process. So, I would prefer to avoid getting the errors in > > the first place. Failing that, I would like to capture the errors and just > > retry the files which failed. > > > > Any help much appreciated. > > > > Regards, > > > > Stuart > > > > > > > > This message contains information that may be privileged or confidential > > and is the property of the Capgemini Group. It is intended only for the > > person to whom it is addressed. If you are not the intended recipient, you > > are not authorized to read, print, retain, copy, disseminate, distribute, > > or use this message or any part thereof. If you receive this message in > > error, please notify the sender immediately and delete all copies of this > > message. > > > > _______________________________________________ > > General mailing list > > General@developer.marklogic.com<mailto:General@developer.marklogic.com> > > Manage your subscription at: > > http://developer.marklogic.com/mailman/listinfo/general > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > > http://developer.marklogic.com/pipermail/general/attachments/20160929/e5f311a1/attachment.html > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: image001.gif > > Type: image/gif > > Size: 1616 bytes > > Desc: image001.gif > > Url : > > http://developer.marklogic.com/pipermail/general/attachments/20160929/e5f311a1/attachment.gif > > > > ------------------------------ > > > > _______________________________________________ > > General mailing list > > General@developer.marklogic.com > > Manage your subscription at: > > http://developer.marklogic.com/mailman/listinfo/general > > > > > > End of General Digest, Vol 147, Issue 51 > > **************************************** > > > > > > ------------------------------ > > _______________________________________________ > General mailing list > General@developer.marklogic.com > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general > > > End of General Digest, Vol 147, Issue 53 > **************************************** > _______________________________________________ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general