> Hi Geert, > > I'd like to give first a quick background on what is inside my 'loaded' > database. It only have 1 forest, has 8 collections inside the forest. The > reason why I'm using collections here is because I have 2 collections on > the same namespace: "http://marklogic.com/ofis/fad"; . And for this > particular module, I just need to transform the documents in "fadperson" > collection. So I'm thinking if I only simply use the map:get($content, > "value"), it will read the whole database which will take up time to load. > Or am I thinking the wrong way? >
> Thanks, > Matt > > P.S. Yes, I am thinking of changing the document-uris in the future but just stick to its original for the meantime, also will change the structure of the document > > On Wed, Oct 14, 2015 at 5:08 PM, <[email protected]> > wrote: > >> Send General mailing list submissions to >> [email protected] >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://developer.marklogic.com/mailman/listinfo/general >> or, via email, send a message with subject or body 'help' to >> [email protected] >> >> You can reach the person managing the list at >> [email protected] >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of General digest..." >> >> >> Today's Topics: >> >> 1. Re: MLCP content transformation between database during >> ingestion not working (Geert Josten) >> 2. Re: MLCP content transformation between database during >> ingestion not working (Matet Layosa) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Wed, 14 Oct 2015 08:01:45 +0000 >> From: Geert Josten <[email protected]> >> Subject: Re: [MarkLogic Dev General] MLCP content transformation >> between database during ingestion not working >> To: MarkLogic Developer Discussion <[email protected]> >> Message-ID: <d243d747.b1916%[email protected]> >> Content-Type: text/plain; charset="windows-1252" >> >> Hi Matt, >> >> MLCP will already call the transform for each document separately, so you >> don?t need the for loops. Use something like let $p := map:get($content, >> "value") instead. I also think you don?t really need to rewrite the uri, >> but maybe you prefer different uri?s as well. >> >> About what is actually going wrong: for each document MLCP will be >> processing fadperson docs 1 to 100, and updating the $content map:map with >> ultimately the uri and contents of only the last fadperson result. You will >> end up with just one new file in the final database if I read this >> correctly.. >> >> Cheers, >> Geert >> >> PS: collection() already returns documents, do you don?t need to use >> doc(base-uri()) on its output either.. >> >> From: <[email protected]<mailto: >> [email protected]>> on behalf of Matet Layosa < >> [email protected]<mailto:[email protected]>> >> Reply-To: MarkLogic Developer Discussion <[email protected] >> <mailto:[email protected]>> >> Date: Wednesday, October 14, 2015 at 9:01 AM >> To: "[email protected]<mailto: >> [email protected]>" <[email protected] >> <mailto:[email protected]>> >> Subject: [MarkLogic Dev General] MLCP content transformation between >> database during ingestion not working >> >> Hi there! I am just new to Marklogic/Xquery and is currently working on a >> project where I need to transform the files from an existing database with >> millions of xml documents, change each files' document structure while >> ingesting it to another database. >> So far, I have this transform module: >> >> xquery version "1.0-ml"; >> module namespace master-person = "http://marklogic.com/ofis/masterperson >> "; >> declare namespace fad = "http://marklogic.com/ofis/fad";; >> >> declare variable $person := /fad:fadperson; >> >> declare function master-person:transform( >> $content as map:map, >> $context as map:map >> ) as map:map* >> { >> for $docs in fn:collection("fadperson") [1 to 100] >> for $p in fn:doc(fn:base-uri($docs))/fad:fadperson >> (: let $map := map:map() :) >> let $id := $p/fad:id/fn:string() >> let $profile := >> <person> >> <id>{$p/fad:id/fn:string()}</id> >> <name> >> <firstname>{$p/fad:first_name/fn:string()}</firstname> >> <lastname>{$p/fad:last_name/fn:string()}</lastname> >> <middlename>{$p/fad:last_name/fn:string()}</middlename> >> >> <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> >> <othername>{$p/fad:other_name/fn:string()}</othername> >> </name> >> <gender>{$p/fad:gender/fn:string()}</gender> >> <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> >> <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> >> <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> >> </person> >> return >> ( >> map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) >> , >> map:put($content, "value", document { $profile }) >> , >> $content >> ) >> >> }; >> >> I also have installed this in my current XDBC server: >> xquery version "1.0-ml"; >> >> xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", >> <options xmlns="xdmp:document-load"> >> <uri>/data/transform-person.xqy</uri> >> <repair>none</repair> >> <permissions>{xdmp:default-permissions()}</permissions> >> </options>) >> >> Then I execute this on terminal: >> mlcp.sh copy -mode local -input_host localhost -input_port 8004 >> -input_username fad-admin -input_password f4d123 -output_host localhost >> -output_port 8041 -output_username fad-admin -input_password f4d123 >> -collection_filter 'fadperson' -transform_module /data/transform-person.xqy >> -transform_namespace "http://marklogic.com/ofis/masterperson"; >> >> the input_port: 8004 = contains the files for transformation; >> output_port:8041 = where the transformed files must be ingested >> >> It doesn't output anything. Neither an error is showing up. It reached >> 100% complete yet no files loaded to the database. >> What am I missing here? Is my transform module code not creating a >> document? >> >> >> Thanks! >> >> Matt >> [email protected]<mailto:[email protected]> >> >> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> http://developer.marklogic.com/pipermail/general/attachments/20151014/b4a27c6f/attachment-0001.html >> >> ------------------------------ >> >> Message: 2 >> Date: Wed, 14 Oct 2015 17:08:51 +0800 >> From: Matet Layosa <[email protected]> >> Subject: Re: [MarkLogic Dev General] MLCP content transformation >> between database during ingestion not working >> To: [email protected] >> Message-ID: >> < >> cadpmupsl-ns3clnkm5sffh1h+begsrunw60+uctwtvbkvet...@mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> >> No problem. They're just a dummy username-password accounts. :) >> >> Hope to hear solution to this problem soon. I'm even thinking of using the >> REST-API transform service but it's not working also. >> >> Thanks, >> Matt >> >> >> On Wed, Oct 14, 2015 at 3:40 PM, <[email protected] >> > >> wrote: >> >> > Send General mailing list submissions to >> > [email protected] >> > >> > To subscribe or unsubscribe via the World Wide Web, visit >> > http://developer.marklogic.com/mailman/listinfo/general >> > or, via email, send a message with subject or body 'help' to >> > [email protected] >> > >> > You can reach the person managing the list at >> > [email protected] >> > >> > When replying, please edit your Subject line so it is more specific >> > than "Re: Contents of General digest..." >> > >> > >> > Today's Topics: >> > >> > 1. MLCP content transformation between database during >> ingestion >> > not working (Matet Layosa) >> > 2. Re: MLCP content transformation between database during >> > ingestion not working (Jeff) >> > >> > >> > ---------------------------------------------------------------------- >> > >> > Message: 1 >> > Date: Wed, 14 Oct 2015 15:01:13 +0800 >> > From: Matet Layosa <[email protected]> >> > Subject: [MarkLogic Dev General] MLCP content transformation between >> > database during ingestion not working >> > To: [email protected] >> > Message-ID: >> > <CADpmUPt= >> > [email protected]> >> > Content-Type: text/plain; charset="utf-8" >> > >> > Hi there! I am just new to Marklogic/Xquery and is currently working on >> a >> > project where I need to transform the files from an existing database >> with >> > millions of xml documents, change each files' document structure while >> > ingesting it to another database. >> > So far, I have this transform module: >> > >> > xquery version "1.0-ml"; >> > >> > module namespace master-person = " >> http://marklogic.com/ofis/masterperson";; >> > >> > declare namespace fad = "http://marklogic.com/ofis/fad";; >> > >> > >> > declare variable $person := /fad:fadperson; >> > >> > >> > declare function master-person:transform( >> > >> > $content as map:map, >> > >> > $context as map:map >> > >> > ) as map:map* >> > >> > { >> > >> > for $docs in fn:collection("fadperson") [1 to 100] >> > >> > for $p in fn:doc(fn:base-uri($docs))/fad:fadperson >> > >> > (: let $map := map:map() :) >> > >> > let $id := $p/fad:id/fn:string() >> > >> > let $profile := >> > >> > <person> >> > >> > <id>{$p/fad:id/fn:string()}</id> >> > >> > <name> >> > >> > <firstname>{$p/fad:first_name/fn:string()}</firstname> >> > >> > <lastname>{$p/fad:last_name/fn:string()}</lastname> >> > >> > <middlename>{$p/fad:last_name/fn:string()}</middlename> >> > >> > >> > <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> >> > >> > <othername>{$p/fad:other_name/fn:string()}</othername> >> > >> > </name> >> > >> > >> > <gender>{$p/fad:gender/fn:string()}</gender> >> > >> > <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> >> > >> > <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> >> > >> > <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> >> > >> > </person> >> > >> > return >> > >> > ( >> > >> > map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) >> > >> > , >> > >> > map:put($content, "value", document { $profile }) >> > >> > , >> > >> > $content >> > >> > ) >> > >> > >> > >> > }; >> > >> > >> > I also have installed this in my current XDBC server: >> > >> > xquery version "1.0-ml"; >> > >> > >> xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", >> > <options xmlns="xdmp:document-load"> >> > <uri>/data/transform-person.xqy</uri> >> > <repair>none</repair> >> > <permissions>{xdmp:default-permissions()}</permissions> >> > </options>) >> > >> > >> > Then I execute this on terminal: >> > mlcp.sh copy -mode local -input_host localhost -input_port 8004 >> > -input_username fad-admin -input_password f4d123 -output_host localhost >> > -output_port 8041 -output_username fad-admin -input_password f4d123 >> > -collection_filter >> > 'fadperson' -transform_module /data/transform-person.xqy >> > -transform_namespace "http://marklogic.com/ofis/masterperson"; >> > >> > the input_port: 8004 = contains the files for transformation; >> > output_port:8041 = where the transformed files must be ingested >> > >> > It doesn't output anything. Neither an error is showing up. It reached >> 100% >> > complete yet no files loaded to the database. >> > What am I missing here? Is my transform module code not creating a >> > document? >> > >> > >> > Thanks! >> > >> > Matt >> > [email protected] >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: >> > >> http://developer.marklogic.com/pipermail/general/attachments/20151014/26dedb1f/attachment-0001.html >> > >> > ------------------------------ >> > >> > Message: 2 >> > Date: Wed, 14 Oct 2015 15:40:36 +0800 >> > From: Jeff <[email protected]> >> > Subject: Re: [MarkLogic Dev General] MLCP content transformation >> > between database during ingestion not working >> > To: MarkLogic Developer Discussion <[email protected]> >> > Message-ID: >> > < >> > cajmd6t+xmt6ofseozfngoqj5xckavygxvk_hlzaxm29dt9z...@mail.gmail.com> >> > Content-Type: text/plain; charset="utf-8" >> > >> > Hi Matt, >> > >> > It might not be wise as practice to include username and password data >> in >> > your posts. Nevertheless, for sure there will be people around here who >> > could provide you with an answer to your question. :) >> > >> > Best regards, >> > Jeff >> > >> > On Wed, Oct 14, 2015 at 3:01 PM, Matet Layosa <[email protected]> >> wrote: >> > >> > > Hi there! I am just new to Marklogic/Xquery and is currently working >> on a >> > > project where I need to transform the files from an existing database >> > with >> > > millions of xml documents, change each files' document structure while >> > > ingesting it to another database. >> > > So far, I have this transform module: >> > > >> > > xquery version "1.0-ml"; >> > > >> > > module namespace master-person = " >> http://marklogic.com/ofis/masterperson >> > "; >> > > >> > > declare namespace fad = "http://marklogic.com/ofis/fad";; >> > > >> > > >> > > declare variable $person := /fad:fadperson; >> > > >> > > >> > > declare function master-person:transform( >> > > >> > > $content as map:map, >> > > >> > > $context as map:map >> > > >> > > ) as map:map* >> > > >> > > { >> > > >> > > for $docs in fn:collection("fadperson") [1 to 100] >> > > >> > > for $p in fn:doc(fn:base-uri($docs))/fad:fadperson >> > > >> > > (: let $map := map:map() :) >> > > >> > > let $id := $p/fad:id/fn:string() >> > > >> > > let $profile := >> > > >> > > <person> >> > > >> > > <id>{$p/fad:id/fn:string()}</id> >> > > >> > > <name> >> > > >> > > <firstname>{$p/fad:first_name/fn:string()}</firstname> >> > > >> > > <lastname>{$p/fad:last_name/fn:string()}</lastname> >> > > >> > > <middlename>{$p/fad:last_name/fn:string()}</middlename> >> > > >> > > >> > > <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> >> > > >> > > <othername>{$p/fad:other_name/fn:string()}</othername> >> > > >> > > </name> >> > > >> > > >> > > <gender>{$p/fad:gender/fn:string()}</gender> >> > > >> > > >> <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> >> > > >> > > <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> >> > > >> > > <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> >> > > >> > > </person> >> > > >> > > return >> > > >> > > ( >> > > >> > > map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) >> > > >> > > , >> > > >> > > map:put($content, "value", document { $profile }) >> > > >> > > , >> > > >> > > $content >> > > >> > > ) >> > > >> > > >> > > >> > > }; >> > > >> > > >> > > I also have installed this in my current XDBC server: >> > > >> > > xquery version "1.0-ml"; >> > > >> > > >> > >> xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", >> > > <options xmlns="xdmp:document-load"> >> > > <uri>/data/transform-person.xqy</uri> >> > > <repair>none</repair> >> > > <permissions>{xdmp:default-permissions()}</permissions> >> > > </options>) >> > > >> > > >> > > Then I execute this on terminal: >> > > mlcp.sh copy -mode local -input_host localhost -input_port 8004 >> > > -input_username fad-admin -input_password f4d123 -output_host >> localhost >> > > -output_port 8041 -output_username fad-admin -input_password f4d123 >> > -collection_filter >> > > 'fadperson' -transform_module /data/transform-person.xqy >> > > -transform_namespace "http://marklogic.com/ofis/masterperson"; >> > > >> > > the input_port: 8004 = contains the files for transformation; >> > > output_port:8041 = where the transformed files must be ingested >> > > >> > > It doesn't output anything. Neither an error is showing up. It reached >> > > 100% complete yet no files loaded to the database. >> > > What am I missing here? Is my transform module code not creating a >> > > document? >> > > >> > > >> > > Thanks! >> > > >> > > Matt >> > > [email protected] >> > > >> > > >> > > >> > > _______________________________________________ >> > > General mailing list >> > > [email protected] >> > > Manage your subscription at: >> > > http://developer.marklogic.com/mailman/listinfo/general >> > > >> > > >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: >> > >> http://developer.marklogic.com/pipermail/general/attachments/20151014/b71f538b/attachment.html >> > >> > ------------------------------ >> > >> > _______________________________________________ >> > General mailing list >> > [email protected] >> > Manage your subscription at: >> > http://developer.marklogic.com/mailman/listinfo/general >> > >> > >> > End of General Digest, Vol 136, Issue 31 >> > **************************************** >> > >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> http://developer.marklogic.com/pipermail/general/attachments/20151014/a7ef7d6a/attachment.html >> >> ------------------------------ >> >> _______________________________________________ >> General mailing list >> [email protected] >> Manage your subscription at: >> http://developer.marklogic.com/mailman/listinfo/general >> >> >> End of General Digest, Vol 136, Issue 32 >> **************************************** >> > >
_______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
