Hi Geert, I'd like to give first a quick background on what is inside my 'loaded' database. It only have 1 forest, has 8 collections inside the forest. The reason why I'm using collections here is because I have 2 collections on the same namespace: "http://marklogic.com/ofis/fad"; . And for this particular module, I just need to transform the documents in "fadperson" collection. So I'm thinking if I only simply use the map:get($content, "value"), it will read the whole database which will take up time to load. Or am I thinking the wrong way?
Thanks, Matt On Wed, Oct 14, 2015 at 5:08 PM, <[email protected]> wrote: > Send General mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://developer.marklogic.com/mailman/listinfo/general > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of General digest..." > > > Today's Topics: > > 1. Re: MLCP content transformation between database during > ingestion not working (Geert Josten) > 2. Re: MLCP content transformation between database during > ingestion not working (Matet Layosa) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 14 Oct 2015 08:01:45 +0000 > From: Geert Josten <[email protected]> > Subject: Re: [MarkLogic Dev General] MLCP content transformation > between database during ingestion not working > To: MarkLogic Developer Discussion <[email protected]> > Message-ID: <d243d747.b1916%[email protected]> > Content-Type: text/plain; charset="windows-1252" > > Hi Matt, > > MLCP will already call the transform for each document separately, so you > don?t need the for loops. Use something like let $p := map:get($content, > "value") instead. I also think you don?t really need to rewrite the uri, > but maybe you prefer different uri?s as well. > > About what is actually going wrong: for each document MLCP will be > processing fadperson docs 1 to 100, and updating the $content map:map with > ultimately the uri and contents of only the last fadperson result. You will > end up with just one new file in the final database if I read this > correctly.. > > Cheers, > Geert > > PS: collection() already returns documents, do you don?t need to use > doc(base-uri()) on its output either.. > > From: <[email protected]<mailto: > [email protected]>> on behalf of Matet Layosa < > [email protected]<mailto:[email protected]>> > Reply-To: MarkLogic Developer Discussion <[email protected] > <mailto:[email protected]>> > Date: Wednesday, October 14, 2015 at 9:01 AM > To: "[email protected]<mailto: > [email protected]>" <[email protected]<mailto: > [email protected]>> > Subject: [MarkLogic Dev General] MLCP content transformation between > database during ingestion not working > > Hi there! I am just new to Marklogic/Xquery and is currently working on a > project where I need to transform the files from an existing database with > millions of xml documents, change each files' document structure while > ingesting it to another database. > So far, I have this transform module: > > xquery version "1.0-ml"; > module namespace master-person = "http://marklogic.com/ofis/masterperson";; > declare namespace fad = "http://marklogic.com/ofis/fad";; > > declare variable $person := /fad:fadperson; > > declare function master-person:transform( > $content as map:map, > $context as map:map > ) as map:map* > { > for $docs in fn:collection("fadperson") [1 to 100] > for $p in fn:doc(fn:base-uri($docs))/fad:fadperson > (: let $map := map:map() :) > let $id := $p/fad:id/fn:string() > let $profile := > <person> > <id>{$p/fad:id/fn:string()}</id> > <name> > <firstname>{$p/fad:first_name/fn:string()}</firstname> > <lastname>{$p/fad:last_name/fn:string()}</lastname> > <middlename>{$p/fad:last_name/fn:string()}</middlename> > > <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> > <othername>{$p/fad:other_name/fn:string()}</othername> > </name> > <gender>{$p/fad:gender/fn:string()}</gender> > <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> > <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> > <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> > </person> > return > ( > map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) > , > map:put($content, "value", document { $profile }) > , > $content > ) > > }; > > I also have installed this in my current XDBC server: > xquery version "1.0-ml"; > > xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", > <options xmlns="xdmp:document-load"> > <uri>/data/transform-person.xqy</uri> > <repair>none</repair> > <permissions>{xdmp:default-permissions()}</permissions> > </options>) > > Then I execute this on terminal: > mlcp.sh copy -mode local -input_host localhost -input_port 8004 > -input_username fad-admin -input_password f4d123 -output_host localhost > -output_port 8041 -output_username fad-admin -input_password f4d123 > -collection_filter 'fadperson' -transform_module /data/transform-person.xqy > -transform_namespace "http://marklogic.com/ofis/masterperson"; > > the input_port: 8004 = contains the files for transformation; > output_port:8041 = where the transformed files must be ingested > > It doesn't output anything. Neither an error is showing up. It reached > 100% complete yet no files loaded to the database. > What am I missing here? Is my transform module code not creating a > document? > > > Thanks! > > Matt > [email protected]<mailto:[email protected]> > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://developer.marklogic.com/pipermail/general/attachments/20151014/b4a27c6f/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Wed, 14 Oct 2015 17:08:51 +0800 > From: Matet Layosa <[email protected]> > Subject: Re: [MarkLogic Dev General] MLCP content transformation > between database during ingestion not working > To: [email protected] > Message-ID: > < > cadpmupsl-ns3clnkm5sffh1h+begsrunw60+uctwtvbkvet...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > No problem. They're just a dummy username-password accounts. :) > > Hope to hear solution to this problem soon. I'm even thinking of using the > REST-API transform service but it's not working also. > > Thanks, > Matt > > > On Wed, Oct 14, 2015 at 3:40 PM, <[email protected]> > wrote: > > > Send General mailing list submissions to > > [email protected] > > > > To subscribe or unsubscribe via the World Wide Web, visit > > http://developer.marklogic.com/mailman/listinfo/general > > or, via email, send a message with subject or body 'help' to > > [email protected] > > > > You can reach the person managing the list at > > [email protected] > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of General digest..." > > > > > > Today's Topics: > > > > 1. MLCP content transformation between database during ingestion > > not working (Matet Layosa) > > 2. Re: MLCP content transformation between database during > > ingestion not working (Jeff) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Wed, 14 Oct 2015 15:01:13 +0800 > > From: Matet Layosa <[email protected]> > > Subject: [MarkLogic Dev General] MLCP content transformation between > > database during ingestion not working > > To: [email protected] > > Message-ID: > > <CADpmUPt= > > [email protected]> > > Content-Type: text/plain; charset="utf-8" > > > > Hi there! I am just new to Marklogic/Xquery and is currently working on a > > project where I need to transform the files from an existing database > with > > millions of xml documents, change each files' document structure while > > ingesting it to another database. > > So far, I have this transform module: > > > > xquery version "1.0-ml"; > > > > module namespace master-person = "http://marklogic.com/ofis/masterperson > "; > > > > declare namespace fad = "http://marklogic.com/ofis/fad";; > > > > > > declare variable $person := /fad:fadperson; > > > > > > declare function master-person:transform( > > > > $content as map:map, > > > > $context as map:map > > > > ) as map:map* > > > > { > > > > for $docs in fn:collection("fadperson") [1 to 100] > > > > for $p in fn:doc(fn:base-uri($docs))/fad:fadperson > > > > (: let $map := map:map() :) > > > > let $id := $p/fad:id/fn:string() > > > > let $profile := > > > > <person> > > > > <id>{$p/fad:id/fn:string()}</id> > > > > <name> > > > > <firstname>{$p/fad:first_name/fn:string()}</firstname> > > > > <lastname>{$p/fad:last_name/fn:string()}</lastname> > > > > <middlename>{$p/fad:last_name/fn:string()}</middlename> > > > > > > <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> > > > > <othername>{$p/fad:other_name/fn:string()}</othername> > > > > </name> > > > > > > <gender>{$p/fad:gender/fn:string()}</gender> > > > > <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> > > > > <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> > > > > <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> > > > > </person> > > > > return > > > > ( > > > > map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) > > > > , > > > > map:put($content, "value", document { $profile }) > > > > , > > > > $content > > > > ) > > > > > > > > }; > > > > > > I also have installed this in my current XDBC server: > > > > xquery version "1.0-ml"; > > > > > xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", > > <options xmlns="xdmp:document-load"> > > <uri>/data/transform-person.xqy</uri> > > <repair>none</repair> > > <permissions>{xdmp:default-permissions()}</permissions> > > </options>) > > > > > > Then I execute this on terminal: > > mlcp.sh copy -mode local -input_host localhost -input_port 8004 > > -input_username fad-admin -input_password f4d123 -output_host localhost > > -output_port 8041 -output_username fad-admin -input_password f4d123 > > -collection_filter > > 'fadperson' -transform_module /data/transform-person.xqy > > -transform_namespace "http://marklogic.com/ofis/masterperson"; > > > > the input_port: 8004 = contains the files for transformation; > > output_port:8041 = where the transformed files must be ingested > > > > It doesn't output anything. Neither an error is showing up. It reached > 100% > > complete yet no files loaded to the database. > > What am I missing here? Is my transform module code not creating a > > document? > > > > > > Thanks! > > > > Matt > > [email protected] > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > > > http://developer.marklogic.com/pipermail/general/attachments/20151014/26dedb1f/attachment-0001.html > > > > ------------------------------ > > > > Message: 2 > > Date: Wed, 14 Oct 2015 15:40:36 +0800 > > From: Jeff <[email protected]> > > Subject: Re: [MarkLogic Dev General] MLCP content transformation > > between database during ingestion not working > > To: MarkLogic Developer Discussion <[email protected]> > > Message-ID: > > < > > cajmd6t+xmt6ofseozfngoqj5xckavygxvk_hlzaxm29dt9z...@mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Hi Matt, > > > > It might not be wise as practice to include username and password data in > > your posts. Nevertheless, for sure there will be people around here who > > could provide you with an answer to your question. :) > > > > Best regards, > > Jeff > > > > On Wed, Oct 14, 2015 at 3:01 PM, Matet Layosa <[email protected]> > wrote: > > > > > Hi there! I am just new to Marklogic/Xquery and is currently working > on a > > > project where I need to transform the files from an existing database > > with > > > millions of xml documents, change each files' document structure while > > > ingesting it to another database. > > > So far, I have this transform module: > > > > > > xquery version "1.0-ml"; > > > > > > module namespace master-person = " > http://marklogic.com/ofis/masterperson > > "; > > > > > > declare namespace fad = "http://marklogic.com/ofis/fad";; > > > > > > > > > declare variable $person := /fad:fadperson; > > > > > > > > > declare function master-person:transform( > > > > > > $content as map:map, > > > > > > $context as map:map > > > > > > ) as map:map* > > > > > > { > > > > > > for $docs in fn:collection("fadperson") [1 to 100] > > > > > > for $p in fn:doc(fn:base-uri($docs))/fad:fadperson > > > > > > (: let $map := map:map() :) > > > > > > let $id := $p/fad:id/fn:string() > > > > > > let $profile := > > > > > > <person> > > > > > > <id>{$p/fad:id/fn:string()}</id> > > > > > > <name> > > > > > > <firstname>{$p/fad:first_name/fn:string()}</firstname> > > > > > > <lastname>{$p/fad:last_name/fn:string()}</lastname> > > > > > > <middlename>{$p/fad:last_name/fn:string()}</middlename> > > > > > > > > > <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> > > > > > > <othername>{$p/fad:other_name/fn:string()}</othername> > > > > > > </name> > > > > > > > > > <gender>{$p/fad:gender/fn:string()}</gender> > > > > > > > <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> > > > > > > <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> > > > > > > <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> > > > > > > </person> > > > > > > return > > > > > > ( > > > > > > map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) > > > > > > , > > > > > > map:put($content, "value", document { $profile }) > > > > > > , > > > > > > $content > > > > > > ) > > > > > > > > > > > > }; > > > > > > > > > I also have installed this in my current XDBC server: > > > > > > xquery version "1.0-ml"; > > > > > > > > > xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", > > > <options xmlns="xdmp:document-load"> > > > <uri>/data/transform-person.xqy</uri> > > > <repair>none</repair> > > > <permissions>{xdmp:default-permissions()}</permissions> > > > </options>) > > > > > > > > > Then I execute this on terminal: > > > mlcp.sh copy -mode local -input_host localhost -input_port 8004 > > > -input_username fad-admin -input_password f4d123 -output_host localhost > > > -output_port 8041 -output_username fad-admin -input_password f4d123 > > -collection_filter > > > 'fadperson' -transform_module /data/transform-person.xqy > > > -transform_namespace "http://marklogic.com/ofis/masterperson"; > > > > > > the input_port: 8004 = contains the files for transformation; > > > output_port:8041 = where the transformed files must be ingested > > > > > > It doesn't output anything. Neither an error is showing up. It reached > > > 100% complete yet no files loaded to the database. > > > What am I missing here? Is my transform module code not creating a > > > document? > > > > > > > > > Thanks! > > > > > > Matt > > > [email protected] > > > > > > > > > > > > _______________________________________________ > > > General mailing list > > > [email protected] > > > Manage your subscription at: > > > http://developer.marklogic.com/mailman/listinfo/general > > > > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > > > http://developer.marklogic.com/pipermail/general/attachments/20151014/b71f538b/attachment.html > > > > ------------------------------ > > > > _______________________________________________ > > General mailing list > > [email protected] > > Manage your subscription at: > > http://developer.marklogic.com/mailman/listinfo/general > > > > > > End of General Digest, Vol 136, Issue 31 > > **************************************** > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://developer.marklogic.com/pipermail/general/attachments/20151014/a7ef7d6a/attachment.html > > ------------------------------ > > _______________________________________________ > General mailing list > [email protected] > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general > > > End of General Digest, Vol 136, Issue 32 > **************************************** >
_______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
