Can you show your actual code? Justin
-- Justin Makeig Director, Product Management MarkLogic jmak...@marklogic.com > On Sep 29, 2016, at 7:09 AM, Shiv Shankar <shiv.shivshan...@gmail.com> wrote: > > HI Justin, > Thanks for your quick reply. However, I used both > declareUpdate({explicitCommit: true}); and declareUpdate(); Still same issue. > > I am just reading from one collection and updating into another document in a > different collection. I managed with insertDocument, but this over writes the > docs which I don't want. I just want to update the attributes ( it is like > UPDATE SET field1:=value, field2=value in loop in oracle world) > > Note: I used lock/unlock options, but getting no lock on that target record. > I used isolation level as part of eval. Couldn't get through it. > XDMP-CONFLICTINGUPDATES: xdmp.eval("//{explicitCommit: > true}\ndeclareUpdate(); \nvar json = \"/Mark...", {}, <options > xmlns="xdmp:eval"><database>67298589469423423423</database><root>/</root><isolation...</options>) > --Conflicting updates xdmp.nodeReplace(cts.doc(" > > > Any thoughts? > > On Thu, Sep 29, 2016 at 7:55 AM, <general-requ...@developer.marklogic.com> > wrote: > Send General mailing list submissions to > general@developer.marklogic.com > > To subscribe or unsubscribe via the World Wide Web, visit > http://developer.marklogic.com/mailman/listinfo/general > or, via email, send a message with subject or body 'help' to > general-requ...@developer.marklogic.com > > You can reach the person managing the list at > general-ow...@developer.marklogic.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of General digest..." > > > Today's Topics: > > 1. Re: XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace > (Justin Makeig) > 2. Re: mlcp Transaction Errors - SVC-EXTIME and XDMP-NOTXN #CGO# > (Jain, Abhishek) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 28 Sep 2016 21:02:26 +0000 > From: Justin Makeig <justin.mak...@marklogic.com> > Subject: Re: [MarkLogic Dev General] XDMP-CONFLICTINGUPDATES: while > using xdmp.nodeReplace > To: MarkLogic Developer Discussion <general@developer.marklogic.com> > Message-ID: <4136b8ca-113a-4373-b6d3-e54d34f15...@marklogic.com> > Content-Type: text/plain; charset="us-ascii" > > The general pattern for doing updates on persisted JSON documents in > JavaScript is to > 1) get the document > 2) turn it into an object > 3) update the object > 4) replace the document node with the updated object > > Here's an example that updates the `balance` property of every document in > the `accounts` collection: > > declareUpdate(); > for(var doc of fn.collection('accounts')) { > var account = doc.toObject(); > account.balance = account.balance * 1.05; > xdmp.nodeReplace(doc, account); > } > > Note that you also need a declareUpdate(). The transaction is committed (or > rolledback) automatically, so you don't need the xdmp.commit(). > > Justin > > -- > Justin Makeig > Director, Product Management > MarkLogic > jmak...@marklogic.com > > > On Sep 28, 2016, at 1:46 PM, Shiv Shankar <shiv.shivshan...@gmail.com> > > wrote: > > > > Hi Rob, > > Thanks for quick reply, > > > > Here is the sample scenario. > > > > I would be building random data and put in to newObjData in a for loop. > > > > 01.json = {"firstName": "abc", "lastName": "xyz"} > > > > newObjectData.firstName= "Donald"; > > > > for Loop > > { > > xdmp.nodeReplace(cts.doc("/test/01.json"),newObjectData); > > xdmp.commit(); > > } > > > > > > > > > > > > On Wed, Sep 28, 2016 at 4:37 PM, Rob Szkutak <rob.szku...@marklogic.com> > > wrote: > > Hi, > > > > Do you have a reproducible case of this I could look at? > > > > Best, > > Rob > > > > Rob Szkutak > > Senior Consultant > > MarkLogic Corporation > > rob.szku...@marklogic.com > > www.marklogic.com > > > > From: Shiv Shankar [shiv.shivshan...@gmail.com] > > Sent: Wednesday, September 28, 2016 3:35 PM > > To: general@developer.marklogic.com; Rob Szkutak > > Subject: XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace > > > > Hi Rob, > > > > I am getting XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace to > > update JSON attributes in FOR loop, instead of over-writing entire > > document. Any advice. > > > > BTW, I used xdmp.eval, but it is not identifying the java script variables. > > > > Thanks > > ShivShankar. > > > > _______________________________________________ > > General mailing list > > General@developer.marklogic.com > > Manage your subscription at: > > http://developer.marklogic.com/mailman/listinfo/general > > > > > ------------------------------ > > Message: 2 > Date: Thu, 29 Sep 2016 11:54:54 +0000 > From: "Jain, Abhishek" <abhishek.b.j...@capgemini.com> > Subject: Re: [MarkLogic Dev General] mlcp Transaction Errors - > SVC-EXTIME and XDMP-NOTXN #CGO# > To: MarkLogic Developer Discussion <general@developer.marklogic.com> > Message-ID: > > <5a37843f322e824aa0d8359ed88cd01f276ff...@in-hyd-dagnd2.corp.capgemini.com> > > Content-Type: text/plain; charset="utf-8" > > Great , Expected !! As it was pretty optimized MLCP Query. > BTW, addressing the second part of your question about logging errors/failed > files ? > I?m also working on similar stuff, this can be done in three ways ? > > > 1. Modifying MLCP library to write all the failed IDs and corresponding > files in a separate log file > > However that requires additional work(java program or x-query ) program to > create new input files of failed record.(Tedious) > > 2. Second way, we need to write a pre-commit trigger, and attach to > database. In pre-commit action module we can > > Write custom logs. > > 3. Similar module can also be written for MLCP Transform. > > > > All three approaches have limitations , which one to choose depends on what > exactly we want to log. > > In my case it pre-commit solution does well. > > I wonder MLCP doesn?t provide RE-TRY option. > > Thanks and Regards, > [Email_CBE.gif]Abhishek Jain > Associate Consultant > > From: general-boun...@developer.marklogic.com > [mailto:general-boun...@developer.marklogic.com] On Behalf Of Stuart Myles > Sent: Friday, September 23, 2016 8:45 PM > To: MarkLogic Developer Discussion > Subject: Re: [MarkLogic Dev General] mlcp Transaction Errors - SVC-EXTIME and > XDMP-NOTXN #CGO# > > Thanks! This helped me prevent the errors from occurring and - as a bonus - > significantly sped up my ingestion. > > I couldn't use exactly the mlcp command line you suggested, since - in the > version of mlcp I'm using - -input_file_type xml isn't allowed, I had to use > -input_file_type documents instead. Also, my input files don't need to be > split. However, bumping up the threads used (to 30 in my case) made the > transaction / timeout complaints go away. And now I'm ingesting 100,000 > documents in 12 minutes, rather than one hour. Much better! > > Regards, > > Stuart > > > > On Fri, Sep 23, 2016 at 3:34 AM, Jain, Abhishek > <abhishek.b.j...@capgemini.com<mailto:abhishek.b.j...@capgemini.com>> wrote: > Hi Stuart, > > MLCP comes with various options, and can be used in various combinations > depending on the file size, memory available and > Other number of nodes, forest etc. > > If you want to try a quick solution you can try this mlcp command : > mlcp import -host yourhost -port 8000 -username userName -password PASSWORD > -input_file_type xml -input_file_path TempData -thread_count > -thread_count_per_split 3 -batch_size 200 -transaction_size 20 > -max_split_size 33554432 -split_input true > change username, input file type etc accordingly. > It?s always good to use splits and threads when working with huge dataset. > Some performance matrix you can consider while using above mlcp : > > 1. In app server settings you can check if connection time out is set > to 0. > > 2. Default spilt size is 32MB, if you can change -max_split_size > 33554432 ( it take in bytes, if your file is bigger ) > > 3. Make sure split and thread ratio remains 1:2 or 1:3 for example > If your document size is 10 MB, and your split size is 1000,000 (1 MB) then > 10/1 = 10 splits > Then you should create 20 or 30 thread for best CPU utilization. > > 4. The above mlcp does well with 150 Million rows, should work for you > as well. > > 5. I assume you have a nice good RAM > 4GB alteast. > > Thanks and Regards, > [Email_CBE.gif]Abhishek Jain > Associate Consultant > Capgemini India | Hyderabad > > From: > general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> > > [mailto:general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>] > On Behalf Of Stuart Myles > Sent: Thursday, September 22, 2016 11:52 PM > To: MarkLogic Developer Discussion > Subject: [MarkLogic Dev General] mlcp Transaction Errors - SVC-EXTIME and > XDMP-NOTXN > > When I'm loading directories of slightly fewer than 100,000 XML files into a > large MarkLogic instance, I often get timeout and transaction errors. If I > re-run the same directory of files which got those errors, I typically don't > get any errors. > > So, I have a few questions: > > * Can I get prevent the errors from happening in the first place - e.g. by > tuning MarkLogic parameters or altering my use of mlcp? > * If I do get errors, what is the best way to get a report on the files which > failed, so I can retry just those ones? Is the best option for me to write > some code to pick out the errors from the log file? And, if so, am I > guaranteed to get all of the files reported? > > Some Details > > The command line template is > > mlcp.sh import -username {1} -password {2} -host localhost -port {4} > -input_file_path {5} -output_uri_replace \"{6},'{7}'\" > > Sometimes, the imports run just fine. However, often I get a large number of > SVC-EXTIME errors followed by a XDMP-NOTXN error. For example: > > 16/09/22 17:54:03 ERROR mapreduce.ContentWriter: SVC-EXTIME: Time limit > exceeded > 16/09/22 17:54:03 WARN mapreduce.ContentWriter: Failed document > 029ccd8ac3323658277ca28fead7a73d.0.xml in > file:/mnt/ingestion/MarkLogicIngestion/smyles/todo/2014_0005.done/029ccd8ac3323658277ca28fead7a73d.0.xml > 16/09/22 17:54:03 ERROR mapreduce.ContentWriter: SVC-EXTIME: Time limit > exceeded > 16/09/22 17:54:03 WARN mapreduce.ContentWriter: Failed document > 02eb4562784255e249c4ec3ed472f9aa.1.xml in > file:/mnt/ingestion/MarkLogicIngestion/smyles/todo/2014_0005.done/02eb4562784255e249c4ec3ed472f9aa.1.xml > 16/09/22 17:54:04 INFO contentpump.LocalJobRunner: completed 33% > 16/09/22 17:54:21 ERROR mapreduce.ContentWriter: XDMP-NOTXN: No transaction > with identifier 9076269665213828952 > > So far, I'm just rerunning the entire directory again. Most of the time, it > ingests fine on the second attempt. However, I have thousands of these > directories to process. So, I would prefer to avoid getting the errors in the > first place. Failing that, I would like to capture the errors and just retry > the files which failed. > > Any help much appreciated. > > Regards, > > Stuart > > > > This message contains information that may be privileged or confidential and > is the property of the Capgemini Group. It is intended only for the person to > whom it is addressed. If you are not the intended recipient, you are not > authorized to read, print, retain, copy, disseminate, distribute, or use this > message or any part thereof. If you receive this message in error, please > notify the sender immediately and delete all copies of this message. > > _______________________________________________ > General mailing list > General@developer.marklogic.com<mailto:General@developer.marklogic.com> > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://developer.marklogic.com/pipermail/general/attachments/20160929/e5f311a1/attachment.html > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: image001.gif > Type: image/gif > Size: 1616 bytes > Desc: image001.gif > Url : > http://developer.marklogic.com/pipermail/general/attachments/20160929/e5f311a1/attachment.gif > > ------------------------------ > > _______________________________________________ > General mailing list > General@developer.marklogic.com > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general > > > End of General Digest, Vol 147, Issue 51 > **************************************** > _______________________________________________ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general