Re: [MarkLogic Dev General] fragments filter large difference

2018-05-24 Thread Geert Josten
Hi Paul, Optimizing XPath is always tricky. I think the optimizer didn’t recognize that `collection($mycollection)/myelem` and ` collection($mycollection)[./myelem]` are (in terms of index resolution) effectively the same. And if the optimizer didn’t, it is likely that MarkLogic would have to

Re: [MarkLogic Dev General] Not able to write in ErrorLog.txt (Markloglogc 9)

2018-05-03 Thread Geert Josten
ErrorLog.txt is used for system-wide messages only since MarkLogic 9, and app(-server) specific messages are written to _ErrorLog.txt. When running xdmp:log from QConsole on port 8000, look for 8000_ErrorLog.txt. You should find your messages there.. Messages can get a little scattered that

Re: [MarkLogic Dev General] Importing temporal documents with MLCP

2018-03-07 Thread Geert Josten
cument-insert( $temporalCollection, $uri, $doc, map:new(( map:entry("collections", $collections), map:entry("permissions", $permissions), map:entry("quality", $quality) )) ) (: return empty-sequence to let MLCP know

Re: [MarkLogic Dev General] Importing temporal documents with MLCP

2018-02-24 Thread Geert Josten
Hi Hans, Sorry for being late with this reply, hopefully it is still useful to you. This was a non-trivial question though, so I had to poke around in the docs to verify various things.. I think it is possible with MLCP. You can override the document-insert mechanism of MLCP using an MLCP

Re: [MarkLogic Dev General] short question about result of sem:sparql-values

2018-02-20 Thread Geert Josten
Hi Erik, Correct. The function returns a sequence of sem:binding objects. A sem:binding is a special type of map:map, so you can use map functions on them. Here is a working example that returns concrete values: xquery version "1.0-ml"; import module namespace sem =

Re: [MarkLogic Dev General] Unique GUID generation in MarkLogic

2018-02-13 Thread Geert Josten
From: general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> <general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>> on behalf of Geert Josten <geert.jos...@marklogic.com

Re: [MarkLogic Dev General] Unique GUID generation in MarkLogic

2018-02-13 Thread Geert Josten
Hi Abhinav, Can you elaborate on what you mean with ‘unique across environments’? Cheers, Geert From: > on behalf of abhinav mishra > Reply-To:

Re: [MarkLogic Dev General] ML 9 Issues

2018-02-07 Thread Geert Josten
is account login. Checking here with this group if anyone else have seen a similar type of issues and found any solutions. Thanks, Praveen. On Wed, Feb 7, 2018 at 2:23 PM, Geert Josten <geert.jos...@marklogic.com<mailto:geert.jos...@marklogic.com>> wrote: Hi Praveen, Which versio

Re: [MarkLogic Dev General] ML 9 Issues

2018-02-07 Thread Geert Josten
Hi Praveen, Which version are you using specifically? If you are not yet using 9.0-4, could you rerun it with that as well? Cheers, Geert From: > on behalf of Praveen Gontla

Re: [MarkLogic Dev General] Strange issue with xdmp:node-delete

2018-02-01 Thread Geert Josten
I concur that the fact ML seems to stop responding does sound like a deadlock. Looking at cluster status, and inspecting the execution queues might reveal a request that doesn’t seem to return. Your code is not using eval or invoke, though, so i don’t think you can create a deadlock with just

Re: [MarkLogic Dev General] archival strategies in bitemporal data

2018-02-01 Thread Geert Josten
Hi Swayam, Can you elaborate a little more? There is temporal:document-protect, which takes archiving properties. Is that what you are after? Cheers, Geert From: > on behalf of Serious Guy

Re: [MarkLogic Dev General] mlcp for multiple host

2018-02-01 Thread Geert Josten
Hi Vikas, You don’t need to specify multiple hosts. MLCP will read out the list of hosts of the cluster automatically through the connect host, and will distribute the load among them. It is essential though that host names as listed inside MarkLogic work as identification on the network too..

Re: [MarkLogic Dev General] How to define relative path for the URL rewriter setting?

2018-01-26 Thread Geert Josten
Hi Evgeny, Keep in mind that relative means relative to the modules-root. Check the modules-root setting of the app-server you are looking at. Cheers, Geert From: > on behalf of Evgeny Degtyarev

Re: [MarkLogic Dev General] question about transactions

2018-01-24 Thread Geert Josten
The outer query runs in query mode, so runs against the timestamp of initial invocation, causing it to never see the result of sem:rdf-insert. You’d have to put the sem:sparql in an xdmp:eval with different-transaction as well. I also wonder though: what are you trying to do, why trying to

Re: [MarkLogic Dev General] Good Way to Automatical

2018-01-22 Thread Geert Josten
I just wanted to mention you also have manage rest api to setup CPF: http://docs.marklogic.com/REST/management/content-processing-framework-(cpf ) Which is leveraged by ml-gradle for deployment of CPF: https://github.com/marklogic-community/ml-gradle/tree/625e3aaadeb58dfa9f040

Re: [MarkLogic Dev General] Dynamic Faceting in Marklogic

2018-01-22 Thread Geert Josten
Hi Arvind, You can define indexes before or after (or during) uploading data. MarkLogic will automatically start reindexation if necessary. Defining indexes before ingestion is more efficient though. So, if you have a chance of ingesting a sample, deciding on indexes based on that, and then

Re: [MarkLogic Dev General] Regarding Spawn function not working

2018-01-10 Thread Geert Josten
Hi Siva, The xdmp:node functions only work on persisted nodes. Make sure $userPersonalInfo is a reference to something from the database, or use an in-memory update library: https://github.com/ryanjdew/XQuery-XML-Memory-Operations Cheers, Geert From:

Re: [MarkLogic Dev General] Best Approach to Manage "Flags" That Might Change Within a Single Transaction

2017-12-07 Thread Geert Josten
You typically avoid these kinds of issues, by using a schedule that gets a fresh latest view on the data each round, or by orchestrating things from outside of MarkLogic.. You could also consider making http calls to localhost instead of eval. Probably not quicker, but perhaps it feels more

Re: [MarkLogic Dev General] How to query metadata

2017-12-07 Thread Geert Josten
Hi Florent, Hidden metadata is part of the document fragment, but not part of ‘full-text’. If you want to do things with it, you need to add a metadata-field for each of them, and then you can also use a field-range index for range queries on them. So yeah, it is a bit like a non-intrusive

Re: [MarkLogic Dev General] Bug in XSLT and XQuery Reference Guide

2017-11-29 Thread Geert Josten
11/29/17, 11:46 AM, "general-boun...@developer.marklogic.com on behalf >of Geert Josten" <general-boun...@developer.marklogic.com on behalf of >geert.jos...@marklogic.com> wrote: > >Thanks, looks like you are right. > >Can you elaborate o

Re: [MarkLogic Dev General] Bug in XSLT and XQuery Reference Guide

2017-11-29 Thread Geert Josten
Thanks, looks like you are right. Can you elaborate on the multiple expressions? Cheers, Geert On 11/29/17, 5:30 PM, "general-boun...@developer.marklogic.com on behalf of Eliot Kimber" wrote: >I didn¹t see a place to

Re: [MarkLogic Dev General] Multi-Database Architecture

2017-11-29 Thread Geert Josten
small forests and not creating too many forests on a host? Would you use the expected forest size as indicator? (eg. no forest < 1gb) Or would you try to create not more forests than cpu-cores /2 per host? Thanks, Andreas 2017-11-28 12:38 GMT+01:00 Geert Josten <geert.jos...@markl

Re: [MarkLogic Dev General] Cannot install 9.3.1 on CentOS 7

2017-11-28 Thread Geert Josten
Regards, -- Florent Georges H2O Consulting http://h2o.consulting/ On 28 November 2017 at 06:42, Geert Josten wrote: Hi Florent, I think you need glibc.x86_64 as well. I use this in mlvagrant: yum -y install glibc.i686 gdb.x86_64 redhat-lsb.x86_64 cyrus-sasl cyrus-sasl-lib cyr

Re: [MarkLogic Dev General] Multi-Database Architecture

2017-11-28 Thread Geert Josten
GMT+01:00 David Gorbet <david.gor...@marklogic.com<mailto:david.gor...@marklogic.com>>: If these are completely separate use cases please consider completely separate clusters. You can use virtualization to make the hardware work out. On Nov 23, 2017, at 12:04 PM, Geert Josten <geert.

Re: [MarkLogic Dev General] Cannot install 9.3.1 on CentOS 7

2017-11-27 Thread Geert Josten
Hi Florent, I think you need glibc.x86_64 as well. I use this in mlvagrant: yum -y install glibc.i686 gdb.x86_64 redhat-lsb.x86_64 cyrus-sasl cyrus-sasl-lib cyrus-sasl-md5 Cheers, Geert From: > on behalf

Re: [MarkLogic Dev General] How to Do Equivalent of While true() Loop In ML?

2017-11-27 Thread Geert Josten
I think ML does not allow to endlessly re-spawn a task. Probably linked to trigger depth. It is to prevent things from running wild. I would definitely recommend running a schedule. I had to do it that way years ago when I was playing around with a custom queue mechanism:

Re: [MarkLogic Dev General] Multi-Database Architecture

2017-11-23 Thread Geert Josten
Hi Andreas, I think each forest has its own in-memory stand, so if each client has a reasonable amount of data, you’ll need several forests per client anyhow. One or multiple databases wouldn’t matter much in that case. I wouldn’t worry too much about in-memory stands though. Memory is much

Re: [MarkLogic Dev General] Spawned Task Appears to Block Other Threads

2017-11-09 Thread Geert Josten
Hi Eliot, I think you kicked off your watcher job with an HTTP request, and it keeps the port open until it finishes. Only one thread can use the port at the same time. Use a different port for task response traffic, or consider running your watcher as a scheduled task. Not super robust, and

Re: [MarkLogic Dev General] Full-text search and JSON

2017-11-07 Thread Geert Josten
Well, you could give your json doc a root property.. :) { root: { id: 1234, text: ³brown fox² } } Cheers On 11/8/17, 4:03 AM, "general-boun...@developer.marklogic.com on behalf of Will Thompson" wrote: >Hi Rob, >

Re: [MarkLogic Dev General] Document access based on field value

2017-11-07 Thread Geert Josten
Hi Richard, It is usually easiest to build up a few layers of roles. Most flexibility is gained when you create separate read and update roles for each group of documents to which you want to control access separately. You can then use role inheritance to give a user or usergroup-specific role

Re: [MarkLogic Dev General] Set of elements to search for search:search()

2017-10-24 Thread Geert Josten
traint terms, I guess the solution is rather to implement and parse my own grammar, and generating the snippets using CTS... :-( Regards, -- Florent Georges H2O Consulting http://h2o.consulting/ On 24 October 2017 at 09:06, Geert Josten wrote: Hi Florent, Have you considered using rest api’s c

Re: [MarkLogic Dev General] Set of elements to search for search:search()

2017-10-24 Thread Geert Josten
Hi Florent, Have you considered using rest api’s capability to take a structured query, rather than relying on search options? That way you can send in complex custom adhoc queries, including those you are after. Yes, you can give different weights in a field, but as you might guess, that is

Re: [MarkLogic Dev General] Regarding Issue in setting element range index

2017-10-10 Thread Geert Josten
Hi Siva, Make sure the reindexer has completed reindexing before adding the new index.. Cheers, Geert From: > on behalf of "Mani, Sivasubramani (ELS)" >

Re: [MarkLogic Dev General] How To Detect Task Time Limit Exceeded Failures?

2017-10-07 Thread Geert Josten
Hi Eliot, I heard the other day that it should be possible to capture such timeouts with a try catch within the code itself. That gives an extra 10 seconds delay which might be sufficient to send out an alert email, or raise some other flag. After those few extra seconds, the timeout gets

Re: [MarkLogic Dev General] How To Reflect Specific Timezone in Formatted Date Time?

2017-10-02 Thread Geert Josten
Hi Eliot, That is not covered by the XPath standard, from the looks of it: https://www.w3.org/TR/xslt20/#date-picture-string I¹m afraid you will have to glue the timezone name to the date yourself. Consider doing a reverse lookup in this map:map:

Re: [MarkLogic Dev General] Querying DateTime values

2017-09-23 Thread Geert Josten
n:QName("", "datetime") On Fri, Sep 22, 2017 at 8:15 AM, Christopher Hamlin <cbham...@gmail.com<mailto:cbham...@gmail.com>> wrote: I'm not sure what is the real problem. xs:dateTime ('2003-08-01T08:00:00Z') > xs:dateTime (fn:current-date() - xs:dayTimeDuration("P30D&

Re: [MarkLogic Dev General] Querying DateTime values

2017-09-22 Thread Geert Josten
Hi Kari, Looks like you misplaced one of the parentheses. Make sure to wrap the string "2017-09-22T08:00:00Z” in xs:dateTime(..) before you try to substract the duration. In provided query you have the xs dateTime cast wrapping both current-date and the duration. Cheers, Geert From:

Re: [MarkLogic Dev General] Construct a lexicon value?

2017-09-18 Thread Geert Josten
Hi Evan, To my knowledge, no.. Except maybe via UDFs.. http://docs.marklogic.com/guide/app-dev/aggregateUDFs cts:values and cts:value-tuples both take queries and options though. And you can filter the returned values manually too.. Cheers, Geert From:

Re: [MarkLogic Dev General] Create temporary user

2017-09-18 Thread Geert Josten
Could SAML authorization be of use to you? http://docs.marklogic.com/guide/security/external-auth#id_81653 SAML support was added in MarkLogic 9. Cheers, Geert From: > on behalf of Andreas Hubmer

Re: [MarkLogic Dev General] Regarding cts:element-value-query

2017-09-01 Thread Geert Josten
Hi Siva, cts:not-query(cts:element-value-query(xs:QName("myelem"), "")) would exclude empty myelem elements.. Kind regards, Geert From: > on behalf of "Mani, Sivasubramani (ELS)"

Re: [MarkLogic Dev General] xray tests

2017-08-31 Thread Geert Josten
Hi, Could you share some more detail on what is happening inside those tests? Would you be able to isolate which test is the culprit by commenting out each one by one? Cheers, Geert From: > on behalf of

Re: [MarkLogic Dev General] Where is General Documentation for the Task Server App?

2017-08-23 Thread Geert Josten
Hi Eliot, You could be hitting a bug in MarkLogic. It might be worth upgrading to 8.0-7, and seeing if it still happens with that version. A lot of patches and performance improvements have been made since 8.0-3.2.. Cheers, Geert On 8/23/17, 5:47 PM, "general-boun...@developer.marklogic.com on

Re: [MarkLogic Dev General] SPARQL 'SERVICE' ?

2017-08-23 Thread Geert Josten
Hi Norbert, MarkLogic does not support: - 14 Basic Federated Query - SPARQL 1.1 Service Description - SPARQL 1.1 Federated Query That is with pure SPARQL. MarkLogic allows wrapping SPARQL statements in XQuery or SJS code that effectively allow mimicking federated search, and the same

Re: [MarkLogic Dev General] Bug in SPARQL date functions

2017-08-23 Thread Geert Josten
Hi Norbert, I don’t think this is a bug. According to the recommendation, the year function expects an xs:dateTime argument, and matches functionality of fn:year-from-dateTime.. https://www.w3.org/TR/sparql11-query/#func-year Cheers, Geert From:

Re: [MarkLogic Dev General] Getting Impossible Value from count()--why?

2017-08-23 Thread Geert Josten
Hi Eliot, Keep in mind that you pass in item-frequency in cts:element-values, but the default for range constraints is likely fragment-frequency. Did you pass in an item-frequency facet-option in there too? Kind regards, Geert On 8/22/17, 10:47 PM, "general-boun...@developer.marklogic.com on

Re: [MarkLogic Dev General] Large job processing question.

2017-08-23 Thread Geert Josten
om: general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> [mailto:general-boun...@developer.marklogic.com] On Behalf Of Geert Josten Sent: August 22, 2017 13:59 To: MarkLogic Developer Discussion <general@developer.marklogic.com<mailto:general@develop

Re: [MarkLogic Dev General] MarkLogic as pubic sparql endpoint?

2017-08-23 Thread Geert Josten
I think I would just create an app-server with a custom rewriter that exposes a custom sparql endpoint only, one that does exactly what you describe. That way you have full control over what is allowed, what data can be read, how results are returned etc. Kind regards, Geert From:

Re: [MarkLogic Dev General] Count of cts:element-values() not equal to number of element instances--what's going on?

2017-08-15 Thread Geert Josten
Wild guess.. Empty prof:overall-elapsed elements, that are ignored/rejected by the range index? Cheers On 8/14/17, 9:58 PM, "general-boun...@developer.marklogic.com on behalf of Eliot Kimber" wrote: >Using both

Re: [MarkLogic Dev General] Date Time value not correct while using DLS API

2017-07-27 Thread Geert Josten
Hi Amit, It is a so-called epoch timestamp, which is calculated as sec or millisec from 1970-01-01. The documentation shows how to convert dateTime to lock timestamps: http://docs.marklogic.com/xdmp:document-locks And this function shows how you could go back:

Re: [MarkLogic Dev General] Binary Document Ingestion in MP4 and MP3 format

2017-07-20 Thread Geert Josten
gic to update the axes timestamps without changing the documents and invoking reindexing. To me it seems that the metadata is connected to the fragment but stored somehow differently. Do you know any more details? Cheers, Andreas 2017-07-20 16:35 GMT+02:00 Geert Josten <geert.jos...@markl

Re: [MarkLogic Dev General] Binary Document Ingestion in MP4 and MP3 format

2017-07-20 Thread Geert Josten
omehow differently. Do you know any more details? Cheers, Andreas 2017-07-20 16:35 GMT+02:00 Geert Josten <geert.jos...@marklogic.com<mailto:geert.jos...@marklogic.com>>: Hi Andreas, I tried to look for a nice Guide section, but couldn’t find one. But there isn’t too much to say about

Re: [MarkLogic Dev General] Binary Document Ingestion in MP4 and MP3 format

2017-07-20 Thread Geert Josten
ion in MP4 and MP3 format Hi Geert, MarkLogic 9 also allows storing simple key/value pairs in hidden document metadata, which is more efficient than document properties I am interested in that new feature. Is there somewhere an explanation how it works (regarding reindexing, ...)? Thanks, An

Re: [MarkLogic Dev General] Binary Document Ingestion in MP4 and MP3 format

2017-07-20 Thread Geert Josten
Pavan <pavan.gu...@soprasteria.com<mailto:pavan.gu...@soprasteria.com>> Date: Thursday, July 20, 2017 at 11:07 AM To: MarkLogic Developer Discussion <general@developer.marklogic.com<mailto:general@developer.marklogic.com>>, Geert Josten <geert.jos...@marklogic.com<mai

Re: [MarkLogic Dev General] Binary Document Ingestion in MP4 and MP3 format

2017-07-20 Thread Geert Josten
Hi Pavan, You can apply xdmp:document-filter on many binary formats, including mp3 and mp4. It will extract meta information like file size and content mime type, and for instance document properties from office documents, and exif tags from images. It will also attempt extract actual text,

Re: [MarkLogic Dev General] Regarding versioning of documents without using DLS.

2017-07-10 Thread Geert Josten
Hi Shabana, I’d recommend looking into the bi-temp functionality, or more specifically the uni-temporal variant that was added in MarkLogic 9. The temporal functionality is embedded much deeper into MarkLogic, and takes away some of the heavy burden of guarding temporal documents are not

Re: [MarkLogic Dev General] RSASHA256 Javascript code for security token to get data from API

2017-07-07 Thread Geert Josten
Hi Nalini, There is nothing MarkLogic specific about this question, so I think this isn't the best place to ask this question. I¹d recommend looking and/or posting a question on StackOverflow, and tagging it with JavaScript. That way you also reach a potentially much bigger community. Kind

Re: [MarkLogic Dev General] MLCP backward compatibility

2017-06-29 Thread Geert Josten
Hi Rajesh, The MLCP guide tells you need MarkLogic 7.0-1 at the least: http://docs.marklogic.com/guide/mlcp/install#id_44231 MLCP relies on a few xqy libraries that should be present server-side, and they were not included in MarkLogic 6 and older. For MarkLogic 6 and before your best option

Re: [MarkLogic Dev General] Jenkins integration with Marklogic roxy deployer framework

2017-06-29 Thread Geert Josten
Hi Santhosh, I think it is easiest to just run Shell commands as build steps, and issue commands like `./ml local bootstrap` etc. If you don’t like adding admin credentials into a deploy/local.properties, you can pass in admin pwd with something like ` --ml.password=`.. Cheers, Geert

Re: [MarkLogic Dev General] MarkLogic 9 XSLT bug(?) with attribute matches

2017-06-23 Thread Geert Josten
Hi Inigo, You are using curly braces inside your XSLT, but your XSLT is in fact literal XML embedded in XQuery, so {local-name()} is interpreted before the xdmp:xslt-eval call. You need to escape those curly braces by doubling them, e.g. {{local-name()}} Cheers, Geert From:

Re: [MarkLogic Dev General] Regarding Marklogic space

2017-06-22 Thread Geert Josten
Hi Siva, It may be wise to reach out to MarkLogic Support as well for more detailed guidance, but I can at least try to explain what the graph is showing you. Your databases have a combined disk footprint of 450 Gb. If deleted fragments have been merged out fully, that is how much the actual

Re: [MarkLogic Dev General] Marklogic - xdmp:filesystem-file

2017-06-22 Thread Geert Josten
Hi Thichxai, You are reading the entire file as a single value. I’d suggest putting bare id’s in your file (no quotes, no commas), one on each line. Then, after reading the file, use fn:tokenize to split on line-end before you pass in the list into your element-value-query.. Kind regards,

Re: [MarkLogic Dev General] Question about bitemporal DB features

2017-06-20 Thread Geert Josten
Hi, MarkLogic will save complete copies of documents, but whether a JSON file of 500Kb on disk will really take a footprint of 500Kb of forest data is rather hard to predict. Values and property names are mapped to a string data table that is stored separately from the structure. If there is a

Re: [MarkLogic Dev General] SJS: use async callback value in the response body

2017-06-20 Thread Geert Josten
Hi Florent, As far as I know, event-driven or async processing simply doesn’t work in SJS.. Cheers, Geert From: > on behalf of Florent Georges > Reply-To:

Re: [MarkLogic Dev General] MLCP Error Return

2017-06-20 Thread Geert Josten
.com>> Subject: Re: [MarkLogic Dev General] MLCP Error Return On Tue, Jun 20, 2017 at 9:25 AM, Geert Josten <geert.jos...@marklogic.com<mailto:geert.jos...@marklogic.com>> wrote: I am not entirely sure about the reasoning behind the logic. It may be to continue processing as many fi

Re: [MarkLogic Dev General] MLCP Error Return

2017-06-20 Thread Geert Josten
Hi Hans, I am not entirely sure about the reasoning behind the logic. It may be to continue processing as many files as possible, without stopping. It may also be that MLCP wasn’t really designed to be used in an embedded way. If you are really looking to automate processing, I think DMSDK

Re: [MarkLogic Dev General] cts:element-value-match for integers

2017-06-19 Thread Geert Josten
The only workaround using the int index that I could think of would be to use ranges with range-queries. Something like: let $pattern := 200 return cts:or-query(( for $i in 0 to 10 let $power := xs:int(math:pow(10, $i)) let $start := $pattern * $power let $end := ($pattern + 1) * $power

Re: [MarkLogic Dev General] MLCP Error Return

2017-06-17 Thread Geert Josten
Hi Hans, Tim, To my knowledge the Java code does return with exit statuses depending on outcome. It looks though they are not properly propagated through mlcp.sh/bat. I’ll see if I can file a bug report for this. In the meantime you could invoking the jar directly according to the rules in the

Re: [MarkLogic Dev General] Accessing properties of in-memory JS object in XQuery

2017-06-16 Thread Geert Josten
It is a json:object (the map:map specialization).. Try: xdmp.xqueryEval( 'declare variable $obj external; map:get($obj, "name")', { obj: {name: 'name', title: 'title' }}); Cheers, Geert On 6/16/17, 9:27 PM, "general-boun...@developer.marklogic.com on behalf of Florent

Re: [MarkLogic Dev General] Host restart issue after joining cluster

2017-06-14 Thread Geert Josten
Hi Rajesh, It is important that both hosts can see each other. They reach out using the host-names defined within the MarkLogic configuration. Make sure both hosts can see the other using that. It is also important import ports are not blocked. If not mistaking that includes 7998 upto 8002.

Re: [MarkLogic Dev General] Help :- Ingested document enrichment

2017-06-01 Thread Geert Josten
Hi Pavan, To my knowledge, most of these do pretty straight-forward HTTP calls, which should work for other public enrichment sites as well. Doing an HTTP call using xdmp:http-get or xdmp:http-post is for sure the easiest way to integrate, and should work well from inside a pipeline action..

Re: [MarkLogic Dev General] Restore reindex

2017-05-31 Thread Geert Josten
Hi Andreas, I think this is something for support. Can you mail them, or reach out to your local MarkLogic contact? Kind regards, Geert From: > on behalf of Andreas Holzgethan

Re: [MarkLogic Dev General] Mlcp transform to break big aggregate xml file to different invidual files

2017-05-30 Thread Geert Josten
Hi Manoj, Keep in mind MLCP transforms receive one $content map:map, but are allowed to return multiple, each representing a file that needs to be persisted. Just return map:map’s each with a unique `uri` and `value` property. I’d recommend combining that with the aggregate_element option on

Re: [MarkLogic Dev General] Invoke SJS from XQuery

2017-05-29 Thread Geert Josten
http://docs.marklogic.com/xdmp:invoke $path The path of the module to be executed as a string. The path is resolved against the root of the App Server evaluating the query, the Modules directory, or relative to the calling module. The module is considered to be JavaScript if the module path

Re: [MarkLogic Dev General] Clarification :- Binary Document Search

2017-05-29 Thread Geert Josten
at 12:57 PM To: MarkLogic Developer Discussion <general@developer.marklogic.com<mailto:general@developer.marklogic.com>>, Geert Josten <geert.jos...@marklogic.com<mailto:geert.jos...@marklogic.com>> Cc: GUPTA Pavan <pavan.gu...@soprasteria.com<mailto:pavan.gu...@sopras

Re: [MarkLogic Dev General] Clarification :- Binary Document Search

2017-05-29 Thread Geert Josten
Hi Ankur, That is kind of by design. MarkLogic does not search binaries directly. Instead you can apply xdmp:document-filter (which uses a built-in 3rd party library) to scrape about 200 different formats for text and metadata. The result is XHTML, and can be saved in document properties or as

Re: [MarkLogic Dev General] concurrent invocation of xquery ending up with duplicate writes

2017-05-23 Thread Geert Josten
Hi Raghu, The best way to ensure concurrent threads not creating a file at the same uri, *is* by using locks. Here is code and some explanation on how to best do that: http://registry.demo.marklogic.com/package/ml-unique Cheers, Geert From:

Re: [MarkLogic Dev General] Processing Large Number of Docs to Get Statistics

2017-05-23 Thread Geert Josten
Hi Eliot, I¹d consider using taskbot (http://registry.demo.marklogic.com/package/taskbot), and using that in combination with either $tb:OPTIONS-SYNC or $tb:OPTIONS-SYNC-UPDATE. It will make optimal use of the TaskServer of the host on which you initiate the call. It doesn¹t scale endlessly, but

Re: [MarkLogic Dev General] Priorities for queries

2017-05-23 Thread Geert Josten
Hi Oleksii, If you use xdmp:spawn or xdmp:spawn-function, you would be able to use the option. It takes ’normal’ and ‘higher’ as values. These priorities have separate queues and worker threads, so they should interfere less with each other. It might also be worth looking into a way to push

Re: [MarkLogic Dev General] MLCP vs backup/restore

2017-05-16 Thread Geert Josten
Hi Rajesh, I’d expect backup/restore to perform much faster. It essentially makes copies of Forest stands on filesystem level, much different than MLCP. It also includes Journals, and if selected Security data too. Getting backup data off the system might be a different question though, but

Re: [MarkLogic Dev General] MarkLogic 9 Redaction

2017-05-10 Thread Geert Josten
Hi Tulasi, With Flex Rep, you can configure push or pull approach, both driven from CPF if I recall correctly. In both cases you can add your own pipelines to do whatever is needed to get the right stuff replicated in the right way. I think in your case you might need to use push approach,

Re: [MarkLogic Dev General] Values endpoint issue in ROXY

2017-04-21 Thread Geert Josten
I think you have hit this issue: https://github.com/marklogic/roxy/issues/758 That ticket contains a fix.. Cheers, Geert PS: note that using the /roxy/rewriter.xqy means you are running in Roxy hybrid mode. If you intend to use MarkLogic REST api only (not Roxy MVC), consider using the real

Re: [MarkLogic Dev General] json:config for XML schema

2017-04-13 Thread Geert Josten
g a XML Schema into custom strategy. Thanks anyway. From: general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> [mailto:general-boun...@developer.marklogic.com] On Behalf Of Geert Josten Sent: Thursday, April 13, 2017 10:22 AM To: MarkLogic Developer Dis

Re: [MarkLogic Dev General] json:config for XML schema

2017-04-13 Thread Geert Josten
55, Steiner, David J. (LNG-DAY) wrote: Hi Geert, Yes, I’ve looked at “full” – way too verbose and I’m very well aware of JSON is ill-suited for hierarchical content. Thanks, David From:general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> [mailto:general-

Re: [MarkLogic Dev General] json:config for XML schema

2017-04-13 Thread Geert Josten
Hi David, That sounds like a very large xsd. Keep in mind JSON is not very well suited for inline elements. I reckon you looked at the full strategy option of json:config? Rather verbose, but simple, and reliable roundtrip.. Cheers, Geert From:

Re: [MarkLogic Dev General] Sending an HTML Email With Attachments

2017-04-13 Thread Geert Josten
e the two together, but it should contain all detail you need. Kind regards, Geert From: <general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>> on behalf of Geert Josten <geert.jos...@marklogic.com<mailto:geert.jos...@marklogic.com>> R

Re: [MarkLogic Dev General] Sending an HTML Email With Attachments

2017-04-13 Thread Geert Josten
> [mailto:general-boun...@developer.marklogic.com] On Behalf Of Geert Josten Sent: 12 April 2017 20:05 To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Sending an HTML Email With Attachments Hi William, Are you trying to send html with embedded images or such, or just

Re: [MarkLogic Dev General] Sending an HTML Email With Attachments

2017-04-12 Thread Geert Josten
Hi William, Are you trying to send html with embedded images or such, or just a pretty formatted message (in html) with some pdf or other doc as collateral? We use html formatted messages in demo-cat, but I am sure I have also sent an attachment with success. I had trouble figuring out how to

Re: [MarkLogic Dev General] xdmp:parse-dateTime

2017-04-11 Thread Geert Josten
The parse-dateTime function will parse dates before the start of the gregorian calendar, but it won’t really be a gDate. For instance: xdmp:parse-dateTime('[D1] [MN] [Y001]', '15 OCTOBER 1582') - xs:dayTimeDuration("P1D") returns 1582-10-14, but officially there was a jump from oct 4 on the

Re: [MarkLogic Dev General] Regarding Text File

2017-04-11 Thread Geert Josten
Hi Siva, Simplest would be to store them as binary nodes. That causes them to get excluded from universal index, but with files that big that might just be what you need.. Kind regards, Geert From: > on

Re: [MarkLogic Dev General] Scramble production data for testing in ML 8.x

2017-04-04 Thread Geert Josten
Hi Shan, Rather than doing it on ingest, you should do the scrambling on export (note: redaction is an export option as well). Unfortunately, MLCP does not allow transformation at export, but it does allow that on copy. You could write your own transform that obfuscates sensitive data.

Re: [MarkLogic Dev General] Roxy rest deployment error to MarkLogic 7.0-2.3 on RedHat 5.1

2017-03-28 Thread Geert Josten
Hi Loren, Open rest-api/config/properties.xml in some text editor, and remove or comment out the line: merge-metadata Must have been added in ML8+. It is hard to track such subtle changes, and compensate or warn about them all from within Roxy. Would be worth a ticket though at

Re: [MarkLogic Dev General] URI_ID whitespace problems with mlcp

2017-03-23 Thread Geert Josten
that with -uri_id, it only takes xml element and json property names. To be able to do that would require using MLCP transforms.. Kind regards, Geert From: <general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>> on behalf of Geert Josten <geert.jos..

Re: [MarkLogic Dev General] Unfiltered, exact searches

2017-03-23 Thread Geert Josten
Hi Andreas, Sounds like a bug indeed. It is as if it appends a case-insensitive flag despite the ‘exact’, because of the all-lowercase ’new’. Can you tell which version of MarkLogic you are running, and on which architecture? Cheers, Geert From:

Re: [MarkLogic Dev General] potential non-conformance with RFC 3986?

2017-03-23 Thread Geert Josten
This seems to sum up all relevant parts nicely: http://stackoverflow.com/questions/15641694/are-uris-case-insensitive/26196170#26196170 And it seems to confirm your statements. I raised RFE #3921 on your behalf.. Cheers, Geert From:

Re: [MarkLogic Dev General] URI_ID whitespace problems with mlcp

2017-03-22 Thread Geert Josten
Valid points all, but MLCP warns about spaces in header names, and proceeds by converting them to underscores before generating XML out of them. On the other hand, though unlikely nor practical, spaces in property names are allowed in JSON. ;-) Cheers, Geert From:

Re: [MarkLogic Dev General] problem with importing Reduency tuples.

2017-03-22 Thread Geert Josten
If you talk about semantics, you probably mean triples instead of tuples (which is a more generic term). If you use SPARQL to query your RDF data / triples, you don’t need to worry about duplicate triples. The triple/sparql engine will deduplicate for you automatically. Kind regards, Geert

Re: [MarkLogic Dev General] URI_ID whitespace problems with mlcp

2017-03-21 Thread Geert Josten
Hi Lucas, I’d recommend using option files. Put each arg on a separate line in a plain text file. Extension free to pick, extra empty lines are allowed for extra readability. The benefit is that you won’t be bothered by the double escaping of first passing in args to mlcp.sh, which in turn

Re: [MarkLogic Dev General] Regarding Error in Marklogic forests

2017-03-20 Thread Geert Josten
Hi Siva, I think it would be wise to reach out to support for this. Cheers, Geert From: > on behalf of "Mani, Sivasubramani (ELS)" > Reply-To: MarkLogic

Re: [MarkLogic Dev General] Using RegEx in xQuery

2017-03-20 Thread Geert Josten
You may want to unwrap entity:entity and suppress entity:entityattr instead, but otherwise this should work just fine all the way down to at least MarkLogic 5.. :) Cheers From: > on behalf of Christopher

Re: [MarkLogic Dev General] Error 500 when queryinq with Curl

2017-03-19 Thread Geert Josten
ser user:pwd -d @"./config.xml" \ -H "Content-type: application/xml" \ http://localhost:8002/LATEST/rest-apis After creating the rest-api in a different t port, using the existing database, I still get the same error. ] Could you please tell me more details on how t

  1   2   3   4   5   6   7   8   9   10   >