Hi Matt, Sorry, I mentioned `-copy_collections` before, but I really meant `-collection_filter` instead. You are correct about a,b,c,d. Not sure what you mean exactly with e, but the transform should be loaded in the modules database that is associated with the output host:port app-server. And it is hence evaluated on that host..
Cheers, Geert From: <[email protected]<mailto:[email protected]>> on behalf of Matet Layosa <[email protected]<mailto:[email protected]>> Reply-To: MarkLogic Developer Discussion <[email protected]<mailto:[email protected]>> Date: Thursday, October 15, 2015 at 3:49 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [MarkLogic Dev General] MLCP content transformation between database during ingestion not working Hi Geert, Okay, now things seem to make sense for me here with how transform works. But I'd like to clarify things on mlcp options. This is what I think I understand base on the documentation: a.) -copy_collections = its default value is "true" so I did not need to specify it. b.) -collection_filter = I used this to exports only documents in the specified collection/s c.) -output_host/username/password = it is the server details where the output result of the transform will be save d.) -input_host/username/password = server details as a source of documents to copy e.) while I run the custom transform load into the output database in my query console. *I'm not sure with the last part (e.) if that's really where it must be loaded :) Anyway, I'll modify the code for transformation just as what you suggest. :) Thanks! Matt Maria Theresa R. Layosa Advanced Science and Technology Institute UP Technology Park Complex, C.P. Garcia Avenue, Diliman, Quezon City, Philippines 1101 Email: [email protected]<mailto:[email protected]> ; [email protected]<mailto:[email protected]> ; On Wed, Oct 14, 2015 at 6:14 PM, <[email protected]<mailto:[email protected]>> wrote: Send General mailing list submissions to [email protected]<mailto:[email protected]> To subscribe or unsubscribe via the World Wide Web, visit http://developer.marklogic.com/mailman/listinfo/general or, via email, send a message with subject or body 'help' to [email protected]<mailto:[email protected]> You can reach the person managing the list at [email protected]<mailto:[email protected]> When replying, please edit your Subject line so it is more specific than "Re: Contents of General digest..." Today's Topics: 1. Re: MLCP content transformation between database during ingestion not working (Geert Josten) ---------------------------------------------------------------------- Message: 1 Date: Wed, 14 Oct 2015 10:14:12 +0000 From: Geert Josten <[email protected]<mailto:[email protected]>> Subject: Re: [MarkLogic Dev General] MLCP content transformation between database during ingestion not working To: MarkLogic Developer Discussion <[email protected]<mailto:[email protected]>> Message-ID: <d243f591.b1a3a%[email protected]<mailto:d243f591.b1a3a%[email protected]>> Content-Type: text/plain; charset="windows-1252" Hi Matt, MLCP transforms still work on a per-document basis. You can either squeeze down the input by using -copy_collections, or add an if statement around the part of the transform function that does the actual job. I just realize that the transform code is executed against the destination. That means you are trying to read collection(?fadperson?) against the destination database. That will return an empty sequence if your destination is empty, causing the entire transform to return empty sequence as well. That explains why you were not seeing any result document at all. My suggestions still stand though, they should get you going.. Cheers From: <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> on behalf of Matet Layosa <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Reply-To: MarkLogic Developer Discussion <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Date: Wednesday, October 14, 2015 at 11:22 AM To: "[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Subject: Re: [MarkLogic Dev General] MLCP content transformation between database during ingestion not working Hi Geert, I'd like to give first a quick background on what is inside my 'loaded' database. It only have 1 forest, has 8 collections inside the forest. The reason why I'm using collections here is because I have 2 collections on the same namespace: "http://marklogic.com/ofis/fad"; . And for this particular module, I just need to transform the documents in "fadperson" collection. So I'm thinking if I only simply use the map:get($content, "value"), it will read the whole database which will take up time to load. Or am I thinking the wrong way? Thanks, Matt On Wed, Oct 14, 2015 at 5:08 PM, <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: Send General mailing list submissions to [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> To subscribe or unsubscribe via the World Wide Web, visit http://developer.marklogic.com/mailman/listinfo/general or, via email, send a message with subject or body 'help' to [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> You can reach the person managing the list at [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> When replying, please edit your Subject line so it is more specific than "Re: Contents of General digest..." Today's Topics: 1. Re: MLCP content transformation between database during ingestion not working (Geert Josten) 2. Re: MLCP content transformation between database during ingestion not working (Matet Layosa) ---------------------------------------------------------------------- Message: 1 Date: Wed, 14 Oct 2015 08:01:45 +0000 From: Geert Josten <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Subject: Re: [MarkLogic Dev General] MLCP content transformation between database during ingestion not working To: MarkLogic Developer Discussion <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Message-ID: <d243d747.b1916%[email protected]<mailto:d243d747.b1916%[email protected]><mailto:d243d747.b1916%[email protected]<mailto:d243d747.b1916%[email protected]>>> Content-Type: text/plain; charset="windows-1252" Hi Matt, MLCP will already call the transform for each document separately, so you don?t need the for loops. Use something like let $p := map:get($content, "value") instead. I also think you don?t really need to rewrite the uri, but maybe you prefer different uri?s as well. About what is actually going wrong: for each document MLCP will be processing fadperson docs 1 to 100, and updating the $content map:map with ultimately the uri and contents of only the last fadperson result. You will end up with just one new file in the final database if I read this correctly.. Cheers, Geert PS: collection() already returns documents, do you don?t need to use doc(base-uri()) on its output either.. From: <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> on behalf of Matet Layosa <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Reply-To: MarkLogic Developer Discussion <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Date: Wednesday, October 14, 2015 at 9:01 AM To: "[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>" <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>>> Subject: [MarkLogic Dev General] MLCP content transformation between database during ingestion not working Hi there! I am just new to Marklogic/Xquery and is currently working on a project where I need to transform the files from an existing database with millions of xml documents, change each files' document structure while ingesting it to another database. So far, I have this transform module: xquery version "1.0-ml"; module namespace master-person = "http://marklogic.com/ofis/masterperson";; declare namespace fad = "http://marklogic.com/ofis/fad";; declare variable $person := /fad:fadperson; declare function master-person:transform( $content as map:map, $context as map:map ) as map:map* { for $docs in fn:collection("fadperson") [1 to 100] for $p in fn:doc(fn:base-uri($docs))/fad:fadperson (: let $map := map:map() :) let $id := $p/fad:id/fn:string() let $profile := <person> <id>{$p/fad:id/fn:string()}</id> <name> <firstname>{$p/fad:first_name/fn:string()}</firstname> <lastname>{$p/fad:last_name/fn:string()}</lastname> <middlename>{$p/fad:last_name/fn:string()}</middlename> <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> <othername>{$p/fad:other_name/fn:string()}</othername> </name> <gender>{$p/fad:gender/fn:string()}</gender> <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> </person> return ( map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) , map:put($content, "value", document { $profile }) , $content ) }; I also have installed this in my current XDBC server: xquery version "1.0-ml"; xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", <options xmlns="xdmp:document-load"> <uri>/data/transform-person.xqy</uri> <repair>none</repair> <permissions>{xdmp:default-permissions()}</permissions> </options>) Then I execute this on terminal: mlcp.sh copy -mode local -input_host localhost -input_port 8004 -input_username fad-admin -input_password f4d123 -output_host localhost -output_port 8041 -output_username fad-admin -input_password f4d123 -collection_filter 'fadperson' -transform_module /data/transform-person.xqy -transform_namespace "http://marklogic.com/ofis/masterperson"; the input_port: 8004 = contains the files for transformation; output_port:8041 = where the transformed files must be ingested It doesn't output anything. Neither an error is showing up. It reached 100% complete yet no files loaded to the database. What am I missing here? Is my transform module code not creating a document? Thanks! Matt [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>><mailto:[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://developer.marklogic.com/pipermail/general/attachments/20151014/b4a27c6f/attachment-0001.html ------------------------------ Message: 2 Date: Wed, 14 Oct 2015 17:08:51 +0800 From: Matet Layosa <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Subject: Re: [MarkLogic Dev General] MLCP content transformation between database during ingestion not working To: [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> Message-ID: <cadpmupsl-ns3clnkm5sffh1h+begsrunw60+uctwtvbkvet...@mail.gmail.com<mailto:cadpmupsl-ns3clnkm5sffh1h%2bbegsrunw60%[email protected]><mailto:cadpmupsl-ns3clnkm5sffh1h%2bbegsrunw60%[email protected]<mailto:cadpmupsl-ns3clnkm5sffh1h%252bbegsrunw60%[email protected]>>> Content-Type: text/plain; charset="utf-8" No problem. They're just a dummy username-password accounts. :) Hope to hear solution to this problem soon. I'm even thinking of using the REST-API transform service but it's not working also. Thanks, Matt On Wed, Oct 14, 2015 at 3:40 PM, <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: > Send General mailing list submissions to > > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > > To subscribe or unsubscribe via the World Wide Web, visit > http://developer.marklogic.com/mailman/listinfo/general > or, via email, send a message with subject or body 'help' to > > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > > You can reach the person managing the list at > > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of General digest..." > > > Today's Topics: > > 1. MLCP content transformation between database during ingestion > not working (Matet Layosa) > 2. Re: MLCP content transformation between database during > ingestion not working (Jeff) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 14 Oct 2015 15:01:13 +0800 > From: Matet Layosa > <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Subject: [MarkLogic Dev General] MLCP content transformation between > database during ingestion not working > To: > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > Message-ID: > <CADpmUPt= > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Content-Type: text/plain; charset="utf-8" > > Hi there! I am just new to Marklogic/Xquery and is currently working on a > project where I need to transform the files from an existing database with > millions of xml documents, change each files' document structure while > ingesting it to another database. > So far, I have this transform module: > > xquery version "1.0-ml"; > > module namespace master-person = "http://marklogic.com/ofis/masterperson";; > > declare namespace fad = "http://marklogic.com/ofis/fad";; > > > declare variable $person := /fad:fadperson; > > > declare function master-person:transform( > > $content as map:map, > > $context as map:map > > ) as map:map* > > { > > for $docs in fn:collection("fadperson") [1 to 100] > > for $p in fn:doc(fn:base-uri($docs))/fad:fadperson > > (: let $map := map:map() :) > > let $id := $p/fad:id/fn:string() > > let $profile := > > <person> > > <id>{$p/fad:id/fn:string()}</id> > > <name> > > <firstname>{$p/fad:first_name/fn:string()}</firstname> > > <lastname>{$p/fad:last_name/fn:string()}</lastname> > > <middlename>{$p/fad:last_name/fn:string()}</middlename> > > > <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> > > <othername>{$p/fad:other_name/fn:string()}</othername> > > </name> > > > <gender>{$p/fad:gender/fn:string()}</gender> > > <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> > > <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> > > <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> > > </person> > > return > > ( > > map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) > > , > > map:put($content, "value", document { $profile }) > > , > > $content > > ) > > > > }; > > > I also have installed this in my current XDBC server: > > xquery version "1.0-ml"; > > xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", > <options xmlns="xdmp:document-load"> > <uri>/data/transform-person.xqy</uri> > <repair>none</repair> > <permissions>{xdmp:default-permissions()}</permissions> > </options>) > > > Then I execute this on terminal: > mlcp.sh copy -mode local -input_host localhost -input_port 8004 > -input_username fad-admin -input_password f4d123 -output_host localhost > -output_port 8041 -output_username fad-admin -input_password f4d123 > -collection_filter > 'fadperson' -transform_module /data/transform-person.xqy > -transform_namespace "http://marklogic.com/ofis/masterperson"; > > the input_port: 8004 = contains the files for transformation; > output_port:8041 = where the transformed files must be ingested > > It doesn't output anything. Neither an error is showing up. It reached 100% > complete yet no files loaded to the database. > What am I missing here? Is my transform module code not creating a > document? > > > Thanks! > > Matt > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://developer.marklogic.com/pipermail/general/attachments/20151014/26dedb1f/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Wed, 14 Oct 2015 15:40:36 +0800 > From: Jeff > <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Subject: Re: [MarkLogic Dev General] MLCP content transformation > between database during ingestion not working > To: MarkLogic Developer Discussion > <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > Message-ID: > < > cajmd6t+xmt6ofseozfngoqj5xckavygxvk_hlzaxm29dt9z...@mail.gmail.com<mailto:cajmd6t%[email protected]><mailto:cajmd6t%[email protected]<mailto:cajmd6t%[email protected]>>> > Content-Type: text/plain; charset="utf-8" > > Hi Matt, > > It might not be wise as practice to include username and password data in > your posts. Nevertheless, for sure there will be people around here who > could provide you with an answer to your question. :) > > Best regards, > Jeff > > On Wed, Oct 14, 2015 at 3:01 PM, Matet Layosa > <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > wrote: > > > Hi there! I am just new to Marklogic/Xquery and is currently working on a > > project where I need to transform the files from an existing database > with > > millions of xml documents, change each files' document structure while > > ingesting it to another database. > > So far, I have this transform module: > > > > xquery version "1.0-ml"; > > > > module namespace master-person = "http://marklogic.com/ofis/masterperson > "; > > > > declare namespace fad = "http://marklogic.com/ofis/fad";; > > > > > > declare variable $person := /fad:fadperson; > > > > > > declare function master-person:transform( > > > > $content as map:map, > > > > $context as map:map > > > > ) as map:map* > > > > { > > > > for $docs in fn:collection("fadperson") [1 to 100] > > > > for $p in fn:doc(fn:base-uri($docs))/fad:fadperson > > > > (: let $map := map:map() :) > > > > let $id := $p/fad:id/fn:string() > > > > let $profile := > > > > <person> > > > > <id>{$p/fad:id/fn:string()}</id> > > > > <name> > > > > <firstname>{$p/fad:first_name/fn:string()}</firstname> > > > > <lastname>{$p/fad:last_name/fn:string()}</lastname> > > > > <middlename>{$p/fad:last_name/fn:string()}</middlename> > > > > > > <nameextension>{$p/fad:name_extension/fn:string()}</nameextension> > > > > <othername>{$p/fad:other_name/fn:string()}</othername> > > > > </name> > > > > > > <gender>{$p/fad:gender/fn:string()}</gender> > > > > <civilstatus>{$p/fad:civil_status/fn:string()}</civilstatus> > > > > <birthdate>{$p/fad:birthdate/fn:string()}</birthdate> > > > > <birthplace>{$p/fad:birthplace/fn:string()}</birthplace> > > > > </person> > > > > return > > > > ( > > > > map:put($content, "uri", fn:concat("/fadperson/", $id, ".xml")) > > > > , > > > > map:put($content, "value", document { $profile }) > > > > , > > > > $content > > > > ) > > > > > > > > }; > > > > > > I also have installed this in my current XDBC server: > > > > xquery version "1.0-ml"; > > > > > xdmp:document-load("/home/jema/Marklogic/apps/data-transform/transform-person.xqy", > > <options xmlns="xdmp:document-load"> > > <uri>/data/transform-person.xqy</uri> > > <repair>none</repair> > > <permissions>{xdmp:default-permissions()}</permissions> > > </options>) > > > > > > Then I execute this on terminal: > > mlcp.sh copy -mode local -input_host localhost -input_port 8004 > > -input_username fad-admin -input_password f4d123 -output_host localhost > > -output_port 8041 -output_username fad-admin -input_password f4d123 > -collection_filter > > 'fadperson' -transform_module /data/transform-person.xqy > > -transform_namespace "http://marklogic.com/ofis/masterperson"; > > > > the input_port: 8004 = contains the files for transformation; > > output_port:8041 = where the transformed files must be ingested > > > > It doesn't output anything. Neither an error is showing up. It reached > > 100% complete yet no files loaded to the database. > > What am I missing here? Is my transform module code not creating a > > document? > > > > > > Thanks! > > > > Matt > > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > > > > > > > > _______________________________________________ > > General mailing list > > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > > Manage your subscription at: > > http://developer.marklogic.com/mailman/listinfo/general > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://developer.marklogic.com/pipermail/general/attachments/20151014/b71f538b/attachment.html > > ------------------------------ > > _______________________________________________ > General mailing list > [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general > > > End of General Digest, Vol 136, Issue 31 > **************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://developer.marklogic.com/pipermail/general/attachments/20151014/a7ef7d6a/attachment.html ------------------------------ _______________________________________________ General mailing list [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general End of General Digest, Vol 136, Issue 32 **************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: http://developer.marklogic.com/pipermail/general/attachments/20151014/d98426fb/attachment.html ------------------------------ _______________________________________________ General mailing list [email protected]<mailto:[email protected]> Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general End of General Digest, Vol 136, Issue 35 ****************************************
_______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
