I see. Setting keep-alive to 0 disables socket pooling for that app server. In my experience, heavy-duty tools that make a large number of small calls, such as Content Pump or Corb, will be unhappy with that and throw the kinds of exceptions you're seeing. I'd suggest trying to set keep-alive back to 5.
Make sure your marklogic-xcc-[version].jar matches the version of MarkLogic you're running. If that doesn't help and you do indeed need keep-alive 0 for certain applications, but keep-alive 5 for others, you might want to consider two dedicated XDBC app servers, one with each setting. Markus On Thu, Oct 15, 2015 at 10:42 PM, Matet Layosa <[email protected]> wrote: > Hi Markus, > > I set my keep-alive timeout to 0 seconds - coz' I've read somewhere in the > documentation to set it to zero when I've been experiencing this error 'Error > parsing HTTP headers: Premature EOF, partial header line read: '' before. > Now, this error doesn't appear anymore, but there's that 'Session has > been closed' error. > > Thanks, > Matt > > > On Fri, Oct 16, 2015 at 9:33 AM, <[email protected]> > wrote: > >> Send General mailing list submissions to >> [email protected] >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://developer.marklogic.com/mailman/listinfo/general >> or, via email, send a message with subject or body 'help' to >> [email protected] >> >> You can reach the person managing the list at >> [email protected] >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of General digest..." >> >> >> Today's Topics: >> >> 1. Re: MLCP content transformation between database during >> ingestion not working (Markus Flatscher) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Thu, 15 Oct 2015 21:33:57 -0400 >> From: Markus Flatscher <[email protected]> >> Subject: Re: [MarkLogic Dev General] MLCP content transformation >> between database during ingestion not working >> To: MarkLogic Developer Discussion <[email protected]> >> Message-ID: >> <CAGgfSV4SRwo3= >> [email protected]> >> Content-Type: text/plain; charset="utf-8" >> >> >> Matet, >> >> what is the keep-alive setting on the XDBC app server you're pointed at? >> >> Markus >> >> On Thu, Oct 15, 2015 at 9:15 PM, Matet Layosa <[email protected]> >> wrote: >> >> > Hi Geert, >> > >> > Yes, I execute my transform module installation in my -output_host >> server. >> > >> > I've also finally made the transformation code running with my expected >> > result. But there's still one thing that I notice during ingestion - >> there >> > are error messages such like: >> > ERROR contentpump.LocalJobRunner: Error committing task: >> > java.lang.IllegalStateException: Session has been closed >> > >> > and also: >> > >> > Cannot obtain connection: Cannot assign requested address >> > >> > I'm not sure if it is a problem regarding my network connection to >> server >> > or if I have misconfiguration on Admin Server >> > >> > Thanks, >> > Matt >> > >> > >> > On Thu, Oct 15, 2015 at 6:08 PM, < >> [email protected]> >> > wrote: >> > >> >> Send General mailing list submissions to >> >> [email protected] >> >> >> >> To subscribe or unsubscribe via the World Wide Web, visit >> >> http://developer.marklogic.com/mailman/listinfo/general >> >> or, via email, send a message with subject or body 'help' to >> >> [email protected] >> >> >> >> You can reach the person managing the list at >> >> [email protected] >> >> >> >> When replying, please edit your Subject line so it is more specific >> >> than "Re: Contents of General digest..." >> >> >> >> >> >> Today's Topics: >> >> >> >> 1. Re: MLCP content transformation between database during >> >> ingestion not working (Geert Josten) >> >> >> >> >> >> ---------------------------------------------------------------------- >> >> >> >> Message: 1 >> >> Date: Thu, 15 Oct 2015 10:08:33 +0000 >> >> From: Geert Josten <[email protected]> >> >> Subject: Re: [MarkLogic Dev General] MLCP content transformation >> >> between database during ingestion not working >> >> To: MarkLogic Developer Discussion <[email protected]> >> >> Message-ID: <d2454716.b2622%[email protected]> >> >> Content-Type: text/plain; charset="us-ascii" >> >> >> >> Hi Matt, >> >> >> >> Sorry, I mentioned `-copy_collections` before, but I really meant >> >> `-collection_filter` instead. You are correct about a,b,c,d. Not sure >> what >> >> you mean exactly with e, but the transform should be loaded in the >> modules >> >> database that is associated with the output host:port app-server. And >> it is >> >> hence evaluated on that host.. >> >> >> >> Cheers, >> >> Geert >> >> >> >> From: <[email protected]<mailto: >> >> [email protected]>> on behalf of Matet Layosa < >> >> [email protected]<mailto:[email protected]>> >> >> Reply-To: MarkLogic Developer Discussion < >> [email protected] >> >> <mailto:[email protected]>> >> >> Date: Thursday, October 15, 2015 at 3:49 AM >> >> To: "[email protected]<mailto: >> >> [email protected]>" <[email protected] >> >> <mailto:[email protected]>> >> >> Subject: Re: [MarkLogic Dev General] MLCP content transformation >> between >> >> database during ingestion not working >> >> >> >> Hi Geert, >> >> >> >> Okay, now things seem to make sense for me here with how transform >> works. >> >> But I'd like to clarify things on mlcp options. This is what I think I >> >> understand base on the documentation: >> >> >> >> a.) -copy_collections = its default value is "true" so I did not need >> >> to specify it. >> >> b.) -collection_filter = I used this to exports only documents in the >> >> specified collection/s >> >> c.) -output_host/username/password = it is the server details where >> >> the output result of the transform will be save >> >> d.) -input_host/username/password = server details as a source of >> >> documents to copy >> >> >> >> e.) while I run the custom transform load into the output database in >> my >> >> query console. >> >> >> >> *I'm not sure with the last part (e.) if that's really where it must be >> >> loaded :) >> >> >> >> Anyway, I'll modify the code for transformation just as what you >> suggest. >> >> :) >> >> >> >> >> >> Thanks! >> >> Matt >> >> >> >> >> >> Maria Theresa R. Layosa >> >> Advanced Science and Technology Institute >> >> UP Technology Park Complex, C.P. Garcia Avenue, >> >> Diliman, Quezon City, Philippines 1101 >> >> Email: [email protected]<mailto:[email protected]> ; >> >> [email protected]<mailto:[email protected]> ; >> >> >> >> On Wed, Oct 14, 2015 at 6:14 PM, < >> [email protected] >> >> <mailto:[email protected]>> wrote: >> >> Send General mailing list submissions to >> >> [email protected]<mailto: >> >> [email protected]> >> >> >> >> To subscribe or unsubscribe via the World Wide Web, visit >> >> http://developer.marklogic.com/mailman/listinfo/general >> >> or, via email, send a message with subject or body 'help' to >> >> [email protected]<mailto: >> >> [email protected]> >> >> >> >> You can reach the person managing the list at >> >> [email protected]<mailto: >> >> [email protected]> >> >> >> >> When replying, please edit your Subject line so it is more specific >> >> than "Re: Contents of General digest..." >> >> >> >> >> >> Today's Topics: >> >> >> >> 1. Re: MLCP content transformation between database during >> >> ingestion not working (Geert Josten) >> >> >> >> >> >> ---------------------------------------------------------------------- >> >> >> >> Message: 1 >> >> Date: Wed, 14 Oct 2015 10:14:12 +0000 >> >> From: Geert Josten <[email protected]<mailto: >> >> [email protected]>> >> >> Subject: Re: [MarkLogic Dev General] MLCP content transformation >> >> between database during ingestion not working >> >> To: MarkLogic Developer Discussion <[email protected] >> >> <mailto:[email protected]>> >> >> Message-ID: <d243f591.b1a3a%[email protected]<mailto: >> >> d243f591.b1a3a%[email protected]>> >> >> Content-Type: text/plain; charset="windows-1252" >> >> >> >> Hi Matt, >> >> >> >> MLCP transforms still work on a per-document basis. You can either >> >> squeeze down the input by using -copy_collections, or add an if >> statement >> >> around the part of the transform function that does the actual job. >> >> >> >> I just realize that the transform code is executed against the >> >> destination. That means you are trying to read collection(?fadperson?) >> >> against the destination database. That will return an empty sequence if >> >> your destination is empty, causing the entire transform to return empty >> >> sequence as well. That explains why you were not seeing any result >> document >> >> at all. My suggestions still stand though, they should get you going.. >> >> >> >> Cheers >> >> >> >> From: <[email protected]<mailto: >> >> [email protected]><mailto: >> >> [email protected]<mailto: >> >> [email protected]>>> on behalf of Matet Layosa < >> >> [email protected]<mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected]>>> >> >> Reply-To: MarkLogic Developer Discussion < >> [email protected] >> >> <mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected] >> >>> >> >> Date: Wednesday, October 14, 2015 at 11:22 AM >> >> To: "[email protected]<mailto: >> >> [email protected]><mailto: >> [email protected] >> >> <mailto:[email protected]>>" < >> >> [email protected]<mailto:[email protected] >> >> ><mailto:[email protected]<mailto: >> >> [email protected]>>> >> >> Subject: Re: [MarkLogic Dev General] MLCP content transformation >> between >> >> database during ingestion not working >> >> >> >> Hi Geert, >> >> >> >> I'd like to give first a quick background on what is inside my 'loaded' >> >> database. It only have 1 forest, has 8 collections inside the forest. >> The >> >> reason why I'm using collections here is because I have 2 collections >> on >> >> the same namespace: "http://marklogic.com/ofis/fad"; . And for this >> >> particular module, I just need to transform the documents in >> "fadperson" >> >> collection. So I'm thinking if I only simply use the map:get($content, >> >> "value"), it will read the whole database which will take up time to >> load. >> >> Or am I thinking the wrong way? >> >> >> >> >> >> Thanks, >> >> Matt >> >> >> >> >> >> On Wed, Oct 14, 2015 at 5:08 PM, < >> [email protected] >> >> <mailto:[email protected]><mailto: >> >> [email protected]<mailto: >> >> [email protected]>>> wrote: >> >> Send General mailing list submissions to >> >> [email protected]<mailto: >> >> [email protected]><mailto: >> [email protected] >> >> <mailto:[email protected]>> >> >> >> >> To subscribe or unsubscribe via the World Wide Web, visit >> >> http://developer.marklogic.com/mailman/listinfo/general >> >> or, via email, send a message with subject or body 'help' to >> >> [email protected]<mailto: >> >> [email protected]><mailto: >> >> [email protected]<mailto: >> >> [email protected]>> >> >> >> >> You can reach the person managing the list at >> >> [email protected]<mailto: >> >> [email protected]><mailto: >> >> [email protected]<mailto: >> >> [email protected]>> >> >> >> >> When replying, please edit your Subject line so it is more specific >> >> than "Re: Contents of General digest..." >> >> >> >> >> >> Today's Topics: >> >> >> >> 1. Re: MLCP content transformation between database during >> >> ingestion not working (Geert Josten) >> >> 2. Re: MLCP content transformation between database during >> >> ingestion not working (Matet Layosa) >> >> >> >> >> >> ---------------------------------------------------------------------- >> >> >> >> Message: 1 >> >> Date: Wed, 14 Oct 2015 08:01:45 +0000 >> >> From: Geert Josten <[email protected]<mailto: >> >> [email protected]><mailto:[email protected]<mailto: >> >> [email protected]>>> >> >> Subject: Re: [MarkLogic Dev General] MLCP content transformation >> >> between database during ingestion not working >> >> To: MarkLogic Developer Discussion <[email protected] >> >> <mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected] >> >>> >> >> Message-ID: <d243d747.b1916%[email protected]<mailto: >> >> d243d747.b1916%[email protected]><mailto: >> >> d243d747.b1916%[email protected]<mailto: >> >> d243d747.b1916%[email protected]>>> >> >> Content-Type: text/plain; charset="windows-1252" >> >> >> >> Hi Matt, >> >> >> >> MLCP will already call the transform for each document separately, so >> you >> >> don?t need the for loops. Use something like let $p := >> map:get($content, >> >> "value") instead. I also think you don?t really need to rewrite the >> uri, >> >> but maybe you prefer different uri?s as well. >> >> >> >> About what is actually going wrong: for each document MLCP will be >> >> processing fadperson docs 1 to 100, and updating the $content map:map >> with >> >> ultimately the uri and contents of only the last fadperson result. You >> will >> >> end up with just one new file in the final database if I read this >> >> correctly.. >> >> >> >> Cheers, >> >> Geert >> >> >> >> PS: collection() already returns documents, do you don?t need to use >> >> doc(base-uri()) on its output either.. >> >> >> >> From: <[email protected]<mailto: >> >> [email protected]><mailto: >> >> [email protected]<mailto: >> >> [email protected]>><mailto: >> >> [email protected]<mailto: >> >> [email protected]><mailto: >> >> [email protected]<mailto: >> >> [email protected]>>>> on behalf of Matet Layosa >> < >> >> [email protected]<mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected]>><mailto: >> >> [email protected]<mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected]>>>> >> >> Reply-To: MarkLogic Developer Discussion < >> [email protected] >> >> <mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected] >> >> >><mailto:[email protected]<mailto: >> >> [email protected]><mailto: >> [email protected] >> >> <mailto:[email protected]>>>> >> >> Date: Wednesday, October 14, 2015 at 9:01 AM >> >> To: "[email protected]<mailto: >> >> [email protected]><mailto: >> [email protected] >> >> <mailto:[email protected]>><mailto: >> >> [email protected]<mailto:[email protected] >> >> ><mailto:[email protected]<mailto: >> >> [email protected]>>>" <[email protected] >> >> <mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected] >> >> >><mailto:[email protected]<mailto: >> >> [email protected]><mailto: >> [email protected] >> >> <mailto:[email protected]>>>> >> >> >> >> Subject: [MarkLogic Dev General] MLCP content transformation between >> >> database during ingestion not working >> >> >> >> Hi there! I am just new to Marklogic/Xquery and is currently working >> on a >> >> project where I need to transform the files from an existing database >> with >> >> millions of xml documents, change each files' document structure while >> >> ingesting it to another database. >> >> So far, I have this transform module: >> >> >> >> xquery version "1.0-ml"; >> >> module namespace master-person = " >> http://marklogic.com/ofis/masterperson >> >> "; >> >> declare namespace fad = "http://marklogic.com/ofis/fad";; >> >> >> >> declare variable $person := /fad:fadperson; >> >> >> >> declare function master-person:transform( >> >> $content as map:map, >> >> $context as map:map >> >> ) as map:map* >> >> { >> >> for $docs in fn:collection("fadperson") [1 to 100] >> >> for $p in fn:doc(fn:base-uri($docs))/fad:fadperson >> >> (: let $map := map:map() :) >> >> let $id := $p/fad:id/fn:string() >> >> let $profile := >> >> <person> >> >> <id>{$p/fad:id/fn:string()}</id> >> >> <name> >> >> <firstname>{$p/fad:first_name/fn:string()}</firstname> >> >> <lastname>{$p/fad:last_name/fn:string()}</lastname> >> >> <middlename>{$p/fad:last_name/fn:string()}</middlename> >> >> >> >> <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> >> >> <othername>{$p/fad:other_name/fn:string()}</othername> >> >> </name> >> >> <gender>{$p/fad:gender/fn:string()}</gender> >> >> >> <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> >> >> <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> >> >> <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> >> >> </person> >> >> return >> >> ( >> >> map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) >> >> , >> >> map:put($content, "value", document { $profile }) >> >> , >> >> $content >> >> ) >> >> >> >> }; >> >> >> >> I also have installed this in my current XDBC server: >> >> xquery version "1.0-ml"; >> >> >> >> >> xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", >> >> <options xmlns="xdmp:document-load"> >> >> <uri>/data/transform-person.xqy</uri> >> >> <repair>none</repair> >> >> <permissions>{xdmp:default-permissions()}</permissions> >> >> </options>) >> >> >> >> Then I execute this on terminal: >> >> mlcp.sh copy -mode local -input_host localhost -input_port 8004 >> >> -input_username fad-admin -input_password f4d123 -output_host localhost >> >> -output_port 8041 -output_username fad-admin -input_password f4d123 >> >> -collection_filter 'fadperson' -transform_module >> /data/transform-person.xqy >> >> -transform_namespace "http://marklogic.com/ofis/masterperson"; >> >> >> >> the input_port: 8004 = contains the files for transformation; >> >> output_port:8041 = where the transformed files must be ingested >> >> >> >> It doesn't output anything. Neither an error is showing up. It reached >> >> 100% complete yet no files loaded to the database. >> >> What am I missing here? Is my transform module code not creating a >> >> document? >> >> >> >> >> >> Thanks! >> >> >> >> Matt >> >> [email protected]<mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected]>><mailto: >> >> [email protected]<mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected]>>> >> >> >> >> >> >> -------------- next part -------------- >> >> An HTML attachment was scrubbed... >> >> URL: >> >> >> http://developer.marklogic.com/pipermail/general/attachments/20151014/b4a27c6f/attachment-0001.html >> >> >> >> ------------------------------ >> >> >> >> Message: 2 >> >> Date: Wed, 14 Oct 2015 17:08:51 +0800 >> >> From: Matet Layosa <[email protected]<mailto:[email protected] >> >> ><mailto:[email protected]<mailto:[email protected]>>> >> >> Subject: Re: [MarkLogic Dev General] MLCP content transformation >> >> between database during ingestion not working >> >> To: [email protected]<mailto: >> >> [email protected]><mailto: >> [email protected] >> >> <mailto:[email protected]>> >> >> Message-ID: >> >> < >> >> cadpmupsl-ns3clnkm5sffh1h+begsrunw60+uctwtvbkvet...@mail.gmail.com >> >> <mailto: >> >> cadpmupsl-ns3clnkm5sffh1h%2bbegsrunw60%[email protected] >> >> ><mailto: >> >> cadpmupsl-ns3clnkm5sffh1h%2bbegsrunw60%[email protected] >> >> <mailto: >> >> >> cadpmupsl-ns3clnkm5sffh1h%252bbegsrunw60%[email protected] >> >> >>> >> >> Content-Type: text/plain; charset="utf-8" >> >> >> >> No problem. They're just a dummy username-password accounts. :) >> >> >> >> Hope to hear solution to this problem soon. I'm even thinking of using >> the >> >> REST-API transform service but it's not working also. >> >> >> >> Thanks, >> >> Matt >> >> >> >> >> >> On Wed, Oct 14, 2015 at 3:40 PM, < >> [email protected] >> >> <mailto:[email protected]><mailto: >> >> [email protected]<mailto: >> >> [email protected]>>> >> >> wrote: >> >> >> >> > Send General mailing list submissions to >> >> > [email protected]<mailto: >> >> [email protected]><mailto: >> [email protected] >> >> <mailto:[email protected]>> >> >> > >> >> > To subscribe or unsubscribe via the World Wide Web, visit >> >> > http://developer.marklogic.com/mailman/listinfo/general >> >> > or, via email, send a message with subject or body 'help' to >> >> > [email protected]<mailto: >> >> [email protected]><mailto: >> >> [email protected]<mailto: >> >> [email protected]>> >> >> > >> >> > You can reach the person managing the list at >> >> > [email protected]<mailto: >> >> [email protected]><mailto: >> >> [email protected]<mailto: >> >> [email protected]>> >> >> > >> >> > When replying, please edit your Subject line so it is more specific >> >> > than "Re: Contents of General digest..." >> >> > >> >> > >> >> > Today's Topics: >> >> > >> >> > 1. MLCP content transformation between database during >> >> ingestion >> >> > not working (Matet Layosa) >> >> > 2. Re: MLCP content transformation between database during >> >> > ingestion not working (Jeff) >> >> > >> >> > >> >> > >> ---------------------------------------------------------------------- >> >> > >> >> > Message: 1 >> >> > Date: Wed, 14 Oct 2015 15:01:13 +0800 >> >> > From: Matet Layosa <[email protected]<mailto:[email protected] >> >> ><mailto:[email protected]<mailto:[email protected]>>> >> >> > Subject: [MarkLogic Dev General] MLCP content transformation between >> >> > database during ingestion not working >> >> > To: [email protected]<mailto: >> >> [email protected]><mailto: >> [email protected] >> >> <mailto:[email protected]>> >> >> > Message-ID: >> >> > <CADpmUPt= >> >> > [email protected]<mailto: >> >> [email protected]><mailto: >> >> [email protected]<mailto: >> >> [email protected]>>> >> >> >> >> > Content-Type: text/plain; charset="utf-8" >> >> > >> >> > Hi there! I am just new to Marklogic/Xquery and is currently working >> on >> >> a >> >> > project where I need to transform the files from an existing database >> >> with >> >> > millions of xml documents, change each files' document structure >> while >> >> > ingesting it to another database. >> >> > So far, I have this transform module: >> >> > >> >> > xquery version "1.0-ml"; >> >> > >> >> > module namespace master-person = " >> >> http://marklogic.com/ofis/masterperson";; >> >> > >> >> > declare namespace fad = "http://marklogic.com/ofis/fad";; >> >> > >> >> > >> >> > declare variable $person := /fad:fadperson; >> >> > >> >> > >> >> > declare function master-person:transform( >> >> > >> >> > $content as map:map, >> >> > >> >> > $context as map:map >> >> > >> >> > ) as map:map* >> >> > >> >> > { >> >> > >> >> > for $docs in fn:collection("fadperson") [1 to 100] >> >> > >> >> > for $p in fn:doc(fn:base-uri($docs))/fad:fadperson >> >> > >> >> > (: let $map := map:map() :) >> >> > >> >> > let $id := $p/fad:id/fn:string() >> >> > >> >> > let $profile := >> >> > >> >> > <person> >> >> > >> >> > <id>{$p/fad:id/fn:string()}</id> >> >> > >> >> > <name> >> >> > >> >> > <firstname>{$p/fad:first_name/fn:string()}</firstname> >> >> > >> >> > <lastname>{$p/fad:last_name/fn:string()}</lastname> >> >> > >> >> > <middlename>{$p/fad:last_name/fn:string()}</middlename> >> >> > >> >> > >> >> > <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> >> >> > >> >> > <othername>{$p/fad:other_name/fn:string()}</othername> >> >> > >> >> > </name> >> >> > >> >> > >> >> > <gender>{$p/fad:gender/fn:string()}</gender> >> >> > >> >> > >> <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> >> >> > >> >> > <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> >> >> > >> >> > <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> >> >> > >> >> > </person> >> >> > >> >> > return >> >> > >> >> > ( >> >> > >> >> > map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) >> >> > >> >> > , >> >> > >> >> > map:put($content, "value", document { $profile }) >> >> > >> >> > , >> >> > >> >> > $content >> >> > >> >> > ) >> >> > >> >> > >> >> > >> >> > }; >> >> > >> >> > >> >> > I also have installed this in my current XDBC server: >> >> > >> >> > xquery version "1.0-ml"; >> >> > >> >> > >> >> >> xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", >> >> > <options xmlns="xdmp:document-load"> >> >> > <uri>/data/transform-person.xqy</uri> >> >> > <repair>none</repair> >> >> > <permissions>{xdmp:default-permissions()}</permissions> >> >> > </options>) >> >> > >> >> > >> >> > Then I execute this on terminal: >> >> > mlcp.sh copy -mode local -input_host localhost -input_port 8004 >> >> > -input_username fad-admin -input_password f4d123 -output_host >> localhost >> >> > -output_port 8041 -output_username fad-admin -input_password f4d123 >> >> > -collection_filter >> >> > 'fadperson' -transform_module /data/transform-person.xqy >> >> > -transform_namespace "http://marklogic.com/ofis/masterperson"; >> >> > >> >> > the input_port: 8004 = contains the files for transformation; >> >> > output_port:8041 = where the transformed files must be ingested >> >> > >> >> > It doesn't output anything. Neither an error is showing up. It >> reached >> >> 100% >> >> > complete yet no files loaded to the database. >> >> > What am I missing here? Is my transform module code not creating a >> >> > document? >> >> > >> >> > >> >> > Thanks! >> >> > >> >> > Matt >> >> > [email protected]<mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected]>> >> >> > -------------- next part -------------- >> >> > An HTML attachment was scrubbed... >> >> > URL: >> >> > >> >> >> http://developer.marklogic.com/pipermail/general/attachments/20151014/26dedb1f/attachment-0001.html >> >> > >> >> > ------------------------------ >> >> > >> >> > Message: 2 >> >> > Date: Wed, 14 Oct 2015 15:40:36 +0800 >> >> > From: Jeff <[email protected]<mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected]>>> >> >> > Subject: Re: [MarkLogic Dev General] MLCP content transformation >> >> > between database during ingestion not working >> >> > To: MarkLogic Developer Discussion <[email protected] >> >> <mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected] >> >>> >> >> > Message-ID: >> >> > < >> >> > cajmd6t+xmt6ofseozfngoqj5xckavygxvk_hlzaxm29dt9z...@mail.gmail.com >> >> <mailto: >> >> cajmd6t%[email protected] >> >> ><mailto: >> >> cajmd6t%[email protected] >> >> <mailto: >> >> cajmd6t%[email protected] >> >>> >> >> > Content-Type: text/plain; charset="utf-8" >> >> > >> >> > Hi Matt, >> >> > >> >> > It might not be wise as practice to include username and password >> data >> >> in >> >> > your posts. Nevertheless, for sure there will be people around here >> who >> >> > could provide you with an answer to your question. :) >> >> > >> >> > Best regards, >> >> > Jeff >> >> > >> >> > On Wed, Oct 14, 2015 at 3:01 PM, Matet Layosa <[email protected] >> >> <mailto:[email protected]><mailto:[email protected]<mailto: >> >> [email protected]>>> wrote: >> >> > >> >> > > Hi there! I am just new to Marklogic/Xquery and is currently >> working >> >> on a >> >> > > project where I need to transform the files from an existing >> database >> >> > with >> >> > > millions of xml documents, change each files' document structure >> while >> >> > > ingesting it to another database. >> >> > > So far, I have this transform module: >> >> > > >> >> > > xquery version "1.0-ml"; >> >> > > >> >> > > module namespace master-person = " >> >> http://marklogic.com/ofis/masterperson >> >> > "; >> >> > > >> >> > > declare namespace fad = "http://marklogic.com/ofis/fad";; >> >> > > >> >> > > >> >> > > declare variable $person := /fad:fadperson; >> >> > > >> >> > > >> >> > > declare function master-person:transform( >> >> > > >> >> > > $content as map:map, >> >> > > >> >> > > $context as map:map >> >> > > >> >> > > ) as map:map* >> >> > > >> >> > > { >> >> > > >> >> > > for $docs in fn:collection("fadperson") [1 to 100] >> >> > > >> >> > > for $p in fn:doc(fn:base-uri($docs))/fad:fadperson >> >> > > >> >> > > (: let $map := map:map() :) >> >> > > >> >> > > let $id := $p/fad:id/fn:string() >> >> > > >> >> > > let $profile := >> >> > > >> >> > > <person> >> >> > > >> >> > > <id>{$p/fad:id/fn:string()}</id> >> >> > > >> >> > > <name> >> >> > > >> >> > > >> <firstname>{$p/fad:first_name/fn:string()}</firstname> >> >> > > >> >> > > <lastname>{$p/fad:last_name/fn:string()}</lastname> >> >> > > >> >> > > >> <middlename>{$p/fad:last_name/fn:string()}</middlename> >> >> > > >> >> > > >> >> > > <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> >> >> > > >> >> > > >> <othername>{$p/fad:other_name/fn:string()}</othername> >> >> > > >> >> > > </name> >> >> > > >> >> > > >> >> > > <gender>{$p/fad:gender/fn:string()}</gender> >> >> > > >> >> > > >> >> <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> >> >> > > >> >> > > <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> >> >> > > >> >> > > >> <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> >> >> > > >> >> > > </person> >> >> > > >> >> > > return >> >> > > >> >> > > ( >> >> > > >> >> > > map:put($content, "uri", fn:concat("/fadperson/", $id, >> ".xml")) >> >> > > >> >> > > , >> >> > > >> >> > > map:put($content, "value", document { $profile }) >> >> > > >> >> > > , >> >> > > >> >> > > $content >> >> > > >> >> > > ) >> >> > > >> >> > > >> >> > > >> >> > > }; >> >> > > >> >> > > >> >> > > I also have installed this in my current XDBC server: >> >> > > >> >> > > xquery version "1.0-ml"; >> >> > > >> >> > > >> >> > >> >> >> xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", >> >> > > <options xmlns="xdmp:document-load"> >> >> > > <uri>/data/transform-person.xqy</uri> >> >> > > <repair>none</repair> >> >> > > <permissions>{xdmp:default-permissions()}</permissions> >> >> > > </options>) >> >> > > >> >> > > >> >> > > Then I execute this on terminal: >> >> > > mlcp.sh copy -mode local -input_host localhost -input_port 8004 >> >> > > -input_username fad-admin -input_password f4d123 -output_host >> >> localhost >> >> > > -output_port 8041 -output_username fad-admin -input_password f4d123 >> >> > -collection_filter >> >> > > 'fadperson' -transform_module /data/transform-person.xqy >> >> > > -transform_namespace "http://marklogic.com/ofis/masterperson"; >> >> > > >> >> > > the input_port: 8004 = contains the files for transformation; >> >> > > output_port:8041 = where the transformed files must be ingested >> >> > > >> >> > > It doesn't output anything. Neither an error is showing up. It >> reached >> >> > > 100% complete yet no files loaded to the database. >> >> > > What am I missing here? Is my transform module code not creating a >> >> > > document? >> >> > > >> >> > > >> >> > > Thanks! >> >> > > >> >> > > Matt >> >> > > [email protected]<mailto:[email protected]><mailto: >> >> [email protected]<mailto:[email protected]>> >> >> > > >> >> > > >> >> > > >> >> > > _______________________________________________ >> >> > > General mailing list >> >> > > [email protected]<mailto: >> >> [email protected]><mailto: >> [email protected] >> >> <mailto:[email protected]>> >> >> > > Manage your subscription at: >> >> > > http://developer.marklogic.com/mailman/listinfo/general >> >> > > >> >> > > >> >> > -------------- next part -------------- >> >> > An HTML attachment was scrubbed... >> >> > URL: >> >> > >> >> >> http://developer.marklogic.com/pipermail/general/attachments/20151014/b71f538b/attachment.html >> >> > >> >> > ------------------------------ >> >> > >> >> > _______________________________________________ >> >> > General mailing list >> >> > [email protected]<mailto: >> [email protected] >> >> ><mailto:[email protected]<mailto: >> >> [email protected]>> >> >> > Manage your subscription at: >> >> > http://developer.marklogic.com/mailman/listinfo/general >> >> > >> >> > >> >> > End of General Digest, Vol 136, Issue 31 >> >> > **************************************** >> >> > >> >> -------------- next part -------------- >> >> An HTML attachment was scrubbed... >> >> URL: >> >> >> http://developer.marklogic.com/pipermail/general/attachments/20151014/a7ef7d6a/attachment.html >> >> >> >> ------------------------------ >> >> >> >> _______________________________________________ >> >> General mailing list >> >> [email protected]<mailto:[email protected] >> >> ><mailto:[email protected]<mailto: >> >> [email protected]>> >> >> Manage your subscription at: >> >> http://developer.marklogic.com/mailman/listinfo/general >> >> >> >> >> >> End of General Digest, Vol 136, Issue 32 >> >> **************************************** >> >> >> >> -------------- next part -------------- >> >> An HTML attachment was scrubbed... >> >> URL: >> >> >> http://developer.marklogic.com/pipermail/general/attachments/20151014/d98426fb/attachment.html >> >> >> >> ------------------------------ >> >> >> >> _______________________________________________ >> >> General mailing list >> >> [email protected]<mailto:[email protected] >> > >> >> Manage your subscription at: >> >> http://developer.marklogic.com/mailman/listinfo/general >> >> >> >> >> >> End of General Digest, Vol 136, Issue 35 >> >> **************************************** >> >> >> >> -------------- next part -------------- >> >> An HTML attachment was scrubbed... >> >> URL: >> >> >> http://developer.marklogic.com/pipermail/general/attachments/20151015/c8a0e9a6/attachment.html >> >> >> >> ------------------------------ >> >> >> >> _______________________________________________ >> >> General mailing list >> >> [email protected] >> >> Manage your subscription at: >> >> http://developer.marklogic.com/mailman/listinfo/general >> >> >> >> >> >> End of General Digest, Vol 136, Issue 38 >> >> **************************************** >> >> >> > >> > >> > _______________________________________________ >> > General mailing list >> > [email protected] >> > Manage your subscription at: >> > http://developer.marklogic.com/mailman/listinfo/general >> > >> > >> >> >> -- >> Markus Flatscher >> Consultant | Avalon Consulting, LLC >> [email protected] >> >> LinkedIn: http://www.linkedin.com/company/avalon-consulting-llc >> Google+: http://www.google.com/+AvalonConsultingLLC >> Twitter: https://twitter.com/avalonconsult >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> http://developer.marklogic.com/pipermail/general/attachments/20151015/ec77c884/attachment.html >> >> ------------------------------ >> >> _______________________________________________ >> General mailing list >> [email protected] >> Manage your subscription at: >> http://developer.marklogic.com/mailman/listinfo/general >> >> >> End of General Digest, Vol 136, Issue 42 >> **************************************** >> > > > _______________________________________________ > General mailing list > [email protected] > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general > > -- Markus Flatscher Consultant | Avalon Consulting, LLC [email protected] LinkedIn: http://www.linkedin.com/company/avalon-consulting-llc Google+: http://www.google.com/+AvalonConsultingLLC Twitter: https://twitter.com/avalonconsult
_______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
