Re: [MarkLogic Dev General] Regarding DMSDK

[Justin Makeig]

Using FilteredForestConfiguration you can provide a white list (“Use only 
these”) or black list (“Don’t use any of these”) of hosts. See 
.

Justin

> On Jan 29, 2018, at 10:38 AM, C. Yaswanth  wrote:
> 
> I want to run the Marklogic Data Movement SDK for transformation in 
> distributed mode on my Marklogic cluster which is running on 3Nodes. Usually 
> in the mlcp we will use the `-host` parameter to specify our hostnames and 
> have `-mode` parameter to define our mode type. Is it possible here in DMSDK 
> to mention all the hostnames of our nodes in ML cluster like this 
> 
> newClient("host1,host2,host3", port, username, password, authentication)
> 
> So that it will distribute the task efficiently. But i didnt seen any 
> documentation of DMSDK with multiple hosts. 
> 
>  1. If i am not giving my all the hostnames then how it gonna distribute the 
> work in parallel fashion.
> 
> Any help is appreciated.
> 
> Thanks
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Exit query without writing to error log?

[Justin Makeig]

If not, we will probably need to restructure the queries to not throw an XQuery 
error at all and just send the appropriate response code and error message. 
Darn, oh well.

As the docs for fn:error() say, the expected behavior when throwing an error is 
to stop the execution, rollback any inflight transactions, and immediately 
return. try/catch can capture (most) errors, but parameter validation sounds 
more like regular program flow than a fatal error condition. Given the 
documented behavior of thrown errors, I’d assert the logging to the error log 
is the correct and expected default behavior. Is there a way you could refactor 
to not throw errors, but return something like a boolean or a status code (that 
you could translate to an HTTP status code on the way out)?

Justin

On Jan 24, 2018, at 10:54 PM, David Ennis 
> wrote:

HI Evan,

Hmm. I just did the same test and was surprised. But yes, still logging the 
error seems to match the documentation.. Sorry about that. My only other 
suggestion is: Is there any common code used as some sort of standard 
invoker/router to the rest of your code? If so, that may be the best place to 
put a tray catch and let the rest bubble up to that point.

Regards,
David

--


From: 
>
 on behalf of Evan Lenz >
Reply-To: MarkLogic Developer Discussion 
>
Date: Thursday, January 25, 2018 at 1:43 AM
To: MarkLogic Developer Discussion 
>
Subject: Re: [MarkLogic Dev General] Exit query without writing to error log?

Hi David,

Following up on this. I tried the custom error handler, and it appears that the 
errors are not actually caught but just forwarded to the error-handler XQuery. 
The impact of this for me (and which is what I'm trying to address) is that the 
errors are still written to the error log, regardless of whether they are 
handled by a custom error handler. Does this accord with your understanding of 
how it's supposed to work?

Do you have any other ideas for how to prevent the writing to the error log? If 
not, we will probably need to restructure the queries to not throw an XQuery 
error at all and just send the appropriate response code and error message. 
Darn, oh well.

Thanks,
Evan

On Wed, Jan 24, 2018 at 2:45 PM, Evan Lenz 
> wrote:
Thanks, guys. David, the custom error handler is a great idea. I had forgotten 
about this option. We're not currently using it but definitely will soon. It 
will help us move forward without fundamentally changing the structure of many 
queries. It will enable us to wrap a virtual try/catch around everything 
without having to change the code itself. Very helpful!

Thanks again,
Evan


On Wed, Jan 24, 2018 at 4:57 AM, David Ennis 
> wrote:
HI.

Further to Florent’s comment, assuming your code is OK with the error bubbling 
up to the top, you can catch them all in a single place and act accordingly  
using a custom error handler. Using this in conjunction with throwing your own 
errors (and handling the code in the top-level handler) can be useful.

https://docs.marklogic.com/guide/app-dev/appserver-control#id_89735

Regards,
David

--


From: 
>
 on behalf of Florent Georges >
Reply-To: MarkLogic Developer Discussion 
>
Date: Wednesday, January 24, 2018 at 1:53 PM
To: MarkLogic Developer Discussion 
>
Subject: Re: [MarkLogic Dev General] Exit query without writing to error log?

Hi Evan,

The error is logged only if it "reaches the top", that is, if it is not caught.

If the error shouldn't behave as such, can't you just catch it before? (and 
yes, potentially return wathever HTTP response you deem appropriate).

Regards,

--
Florent Georges
H2O Consulting
http://h2o.consulting/


On 24 January 2018 at 03:22, Evan Lenz wrote:
Hello,

My client has XQuery parameter-validation code which calls the error() function 
when invalid parameters are supplied. This event gets written to the error log. 
Is there a way to prevent this?

I recognize the best practice is to return a more appropriate HTTP response 
code (than 500) when handling invalid input. I also recognize this is anathema 
to FP. But my question still stands:

Is there a way to interrupt/exit a query without triggering an Error-level log 
message?

Thanks,

Evan Lenz
President, Lenz Consulting Group, Inc.

Re: [MarkLogic Dev General] (no subject)

[Justin Makeig]

You’re trying to update all of the documents in the “input” collection in a 
single transaction, the default scope of a JavaScript module. For small numbers 
of documents (hundreds or thousands) that will work. For large or unknown 
numbers of documents that will generally overwhelm some resource. In your case, 
you’ve blown out your expanded tree cache. Your best bet is to break the 
transformation into multiple transactions and spread those out over multiple 
hosts in a cluster. The MarkLogic Data Movement SDK is designed for exactly 
such workloads. Take a look at the docs 
<https://docs.marklogic.com/guide/java/data-movement#id_51555> for more 
information on how to orchestrate a bulk transformation from Java.

Justin

--
Justin Makeig
Senior Director, Product Management
MarkLogic
jmak...@marklogic.com<mailto:jmak...@marklogic.com>

On Jan 24, 2018, at 10:19 AM, C. Yaswanth 
<rocking...@gmail.com<mailto:rocking...@gmail.com>> wrote:

Hi All,

Actually i have a set of json files(i.e.Total : 1M with Size 500MB). Each json 
file has 18 Keys. I tried to implement Envelope pattern using below Javascript

'use strict';
declareUpdate()
var docs = fn.collection("input");
for(var doc of docs) {
  var transformed = {};

  transformed.Metadata = { "Last Used" : ""};
  transformed.Updated = { "University" : "UCLA"}
  transformed.Source = doc; //Sending original data under Source section
  xdmp.nodeReplace(doc,transformed)
}

I tried invoking this `JS.sjs` using JAVA API of marklogic 9. But i encountered 
below error :

Exception in thread "main" com.marklogic.client.FailedRequestException: 
Local message: failed to apply resource at invoke: Internal Server Error. 
Server Message: XDMP-EXPNTREECACHEFULL: for(var doc of docs) { -- Expanded tree 
cache full on host localhost uri file.json-0-968991
 at 
com.marklogic.client.impl.OkHttpServices.checkStatus(OkHttpServices.java:4317)
 at 
com.marklogic.client.impl.OkHttpServices.postIteratedResourceImpl(OkHttpServices.java:3831)
 at 
com.marklogic.client.impl.OkHttpServices.postEvalInvoke(OkHttpServices.java:3768)
 at 
com.marklogic.client.impl.ServerEvaluationCallImpl.eval(ServerEvaluationCallImpl.java:164)
 at 
com.marklogic.client.impl.ServerEvaluationCallImpl.eval(ServerEvaluationCallImpl.java:153)
 at 
com.marklogic.client.impl.ServerEvaluationCallImpl.evalAs(ServerEvaluationCallImpl.java:144)
 at bulkimport.Tsm.main(Tsm.java:19)

I went through documentation 
(i.e.https://help.marklogic.com/knowledgebase/article/View/9/16/resolving-xdmp-expntreecachefull-errors)
  where they had mentioned ways to resolve this error. Following that i had 
increased `expanded tree cache size*` to `2048` but still i am facing same 
error.

How can i optimize by above code (i.e.`JS.sjs`) to avoid this error ?

Any help is appreciated.
___
General mailing list
General@developer.marklogic.com<mailto:General@developer.marklogic.com>
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Best Approach to Manage "Flags" That Might Change Within a Single Transaction

[Justin Makeig]

<https://docs.marklogic.com/guide/java/data-movement>

The open-source Data Movement SDK that ships with v4 of the Java Client API 
<https://github.com/marklogic/java-client-api> was designed for bulk 
orchestration. It runs in an external Java process and doesn’t rely on the task 
server, CPF, or large transactions. It allows you to break up import (MarkLogic 
as sink), export (MarkLogic as source), or transformation (MarkLogic as source 
and sink) jobs into chunks and parallelize them over a cluster. It handles all 
of tricky threading in Java and gives you nice callback interfaces for progress 
and error handling. It will even handle failover events on the MarkLogic 
cluster over which it’s working.

Justin

--
Justin Makeig
Senior Director, Product Management
MarkLogic
jmak...@marklogic.com<mailto:jmak...@marklogic.com>


On Dec 7, 2017, at 10:34 AM, Will Thompson 
<wthomp...@oconnors.com<mailto:wthomp...@oconnors.com>> wrote:

Hi Eliot,

I have found that trying to control a long batch process through a single 
long-running transaction is too much trouble because of the same issues you are 
having w.r.t. visibility of updates. I think it seems natural at first because 
XQuery is such a nice language, but the accumulation of workarounds for the 
transaction boundary issues can turn into a maintenance nightmare. I recently 
worked on a legacy project like that (a real cautionary tale!). It's much less 
of an uphill battle if you can break the long-running task into multiple 
shorter-running ones, either through CPF, triggers, scheduled tasks, etc. and 
avoid all that. I like to keep projects native when possible and (through lots 
of trial and error) have become accustomed to the constraints, but a lot of 
developers just factor this part out into Java or another language where they 
can execute a long-running controller type loop without any of those headaches.

-Will


On Dec 7, 2017, at 11:19 AM, Eliot Kimber 
<ekim...@contrext.com<mailto:ekim...@contrext.com>> wrote:

I think I've solved my problem by once again being more careful about holding 
elements in memory. By replacing global reads of my job doc with on-demand 
reads through xdmp:eval() I seem to have resolved my issue with changes to the 
job doc not being seen within the same separate transaction (e.g,, my read 
loop). I seem to be unable to let go of my procedural language brain damage

Still, it seems like having a general, cross-application field or shared memory 
mechanism would be useful for this type of application where one app (e.g., my 
Web UI) spawns tasks that do the work and need a way to dynamically communicate 
within the scope of a single long-running transaction. At least that's the way 
I would go about building this type of application in a different environment.

Cheers,

E.
--
Eliot Kimber
https://urldefense.proofpoint.com/v2/url?u=http-3A__contrext.com=DwIGaQ=IdrBOxAMwHPzAikPNzltHw=_thRNTuzvzYaEDwaA_AfnAe5hN2lWgi6qdluz6ApLYI=Ab7G2N7CzquTxNem21cvkP5LWiFn1bSROAHHMlRw3xQ=axTz3BGMQtTK1QP5MhZE5Zq0ANiiWVI0SuRG4dVpxp8=


On 12/7/17, 10:48 AM, 
"general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>
 on behalf of Eliot Kimber" 
<general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>
 on behalf of ekim...@contrext.com<mailto:ekim...@contrext.com>> wrote:

  I don't think server fields are going to work because they are per 
application server and I have different application servers at work.

  There is an HTTP server that gets the pause/resume request and then spawned 
tasks running the TaskServer that need to read the field.

  My experiments show that, per the docs, a field changed by one app is not 
seen by a different app.

  Cheers,

  Eliot
  --
  Eliot Kimber
  
https://urldefense.proofpoint.com/v2/url?u=http-3A__contrext.com=DwIGaQ=IdrBOxAMwHPzAikPNzltHw=_thRNTuzvzYaEDwaA_AfnAe5hN2lWgi6qdluz6ApLYI=Ab7G2N7CzquTxNem21cvkP5LWiFn1bSROAHHMlRw3xQ=axTz3BGMQtTK1QP5MhZE5Zq0ANiiWVI0SuRG4dVpxp8=


  On 12/7/17, 10:13 AM, 
"general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>
 on behalf of Eliot Kimber" 
<general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>
 on behalf of ekim...@contrext.com<mailto:ekim...@contrext.com>> wrote:

  I had not considered server fields--I'll check it out.

  Cheers,

  E.

  --
  Eliot Kimber
  
https://urldefense.proofpoint.com/v2/url?u=http-3A__contrext.com=DwIGaQ=IdrBOxAMwHPzAikPNzltHw=_thRNTuzvzYaEDwaA_AfnAe5hN2lWgi6qdluz6ApLYI=Ab7G2N7CzquTxNem21cvkP5LWiFn1bSROAHHMlRw3xQ=axTz3BGMQtTK1QP5MhZE5Zq0ANiiWVI0SuRG4dVpxp8=


  On 12/7/17, 10:11 AM, 
"general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>
 on behalf of Erik Hennum" 
<general-boun...@developer.marklogic.com&

Re: [MarkLogic Dev General] Retrieving collections based on document uri

[Justin Makeig]

You can build a collection query dynamically from the result sequence of 
collection URIs that xdmp:document-get-collections() returns. For example,

cts:and-query((
  …
  cts:collection-query(xdmp:document-get-collections($uri)),
  …
)…)

On Nov 9, 2017, at 3:17 PM, Pooja Rajput 
> wrote:

Thanks for the response Ron.
I need to use this inside cts-query so that I can filter the type of 
collections as well.
Hence I am looking for similar cts query.

Thanks,
Pooja

From: 
general-boun...@developer.marklogic.com
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of Ron Hitchens
Sent: Thursday, November 9, 2017 4:14 PM
To: MarkLogic Developer Discussion 
>
Subject: Re: [MarkLogic Dev General] Retrieving collections based on document 
uri


https://docs.marklogic.com/xdmp:document-get-collections


Ron Hitchens r...@overstory.co.uk, +44 7879 358212



On November 9, 2017 at 11:12:18 PM, Pooja Rajput 
(pooja.raj...@healthbi.com) wrote:
Hi Team,
I need to retrieve the collections associated with particular document.
All I have is document uri. How can I retrieve this info using cts query like 
xdmp:Document-get-collections provides results.
Any example is appreciated.

Thanks
Pooja


___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general
___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] XQuery in async multidoc operations

[Justin Makeig]

Rather than describing your solution, can you tell us about the problem you’re 
trying to solve? Are you trying to do a bulk transformation?

Justin

--
Justin Makeig
Senior Director, Product Management
MarkLogic
jmak...@marklogic.com

> On Sep 26, 2017, at 10:02 AM, Shmennen <shmen...@yahoo.com> wrote:
> 
> Hello All,
> 
>Is there any option to perform XQueries in asynchronous multidocument 
> operations? Based on doc https://docs.marklogic.com/guide/java/data-movement  
> I cannot find anything useful.
> 
> Thanks
> Paul JP.
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general





___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Error by DM SDK on ML 8.0.6

[Justin Makeig]

The Data Movement SDK depends on MarkLogic 9. We’re looking at how we might 
back-port it to earlier versions, but those aren’t supported in the initial 
release.

Justin

--
Justin Makeig
Senior Director, Product Management
MarkLogic
jmak...@marklogic.com


> On Sep 20, 2017, at 1:23 PM, Shiv Shankar <shiv.shivshan...@gmail.com> wrote:
> 
> Hi,
> I keep getting below error while loading the data from a file into ML 8.x 
> using DataMovement SDK. Same code is working fine with 9.x. Any advice?
> 
> Error.
> 
> Exception in thread "main" com.marklogic.client.ResourceNotFoundException: 
> Local message: Could not read resource at internal/forestinfo. Server 
> Message: Request failed. Error body not received from server
> 
> at 
> com.marklogic.client.impl.JerseyServices.checkStatus(JerseyServices.java:4297)
> 
> 
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general






smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Create temporary user

[Justin Makeig]

Andreas,
Rather than describe your solution, can you explain the problem you’re trying 
to solve? Why do you think you need a temporary user? What permission/privilege 
challenge are you trying to address? 

You might also take a look at amps 
<https://docs.marklogic.com/guide/admin/security#id_81246>. An amp allows a 
security administrator to elevate the privileges of a specific function. This 
is beneficial in that the security is defined in configuration, not code.

Justin


--
Justin Makeig
Senior Director, Product Management
MarkLogic
jmak...@marklogic.com



> On Sep 15, 2017, at 4:29 AM, Andreas Holzgethan 
> <andreas.holzget...@ebcont.com> wrote:
> 
> Hi @all,
> 
> I need the possibility to create temporary user for a transaction.
> I just found in the documentation that such a functionality is used when for 
> example LDAP is configured as an external security.
> 
> Could you please explain me how this is done there?
> 
> My thirst thought was to create a user with the function 
> "sec:create-user-with-role". At the end of the transaction I would just call 
> the function "sec:remove-user".
> Could you please give me feedback about this implementation?
> Is such a implementation a big influence on the performance?
> 
> Thanks!
> 
> Best regards
> Andreas Holzgethan
> 
> Andreas Holzgethan BSc.
> 
> IT Consultant
> 
> 
> EBCONT enterprise technologies GmbH
> Millennium Tower
> Handelskai 94-96
> 1200 Wien
> 
> Mobil: +43 664 606 517 05
> 
> 
> 
> Email:andreas.holzget...@ebcont.com
> Web:http://www.ebcont-et.com/
> 
> OUR TEAM IS YOUR SUCCESS
>
> HG St. Pölten - FN 293731 h
> UID: ATU63444589
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Getting Impossible Value from count()--why?

[Justin Makeig]

You should also take a look at cts:sum-aggregate() 
. Unlike fn:sum(), which pulls 
values from the D-nodes to the E-node where it’s being executed, 
cts:sum-aggregate() allows you to scope the sum calculation by a query and 
delegate the aggregate calculations to the D-nodes in a cluster. The E-node 
needs only sum the partial aggregates. (It uses the underlying map-reduce 
framework available for UDFs 
.) This allows for more 
parallelization and less intermediate data being shuffled among nodes in a 
cluster. All of the aggregate functions in the cts namespace operate this way. 
Although some aggregates are more parallelizable than others.

Justin


> On Aug 23, 2017, at 7:34 AM, Eliot Kimber  wrote:
> 
> Yes, I added “item-frequency” to my cts:element-values() call and now all the 
> numbers appear to be correct.
> 
> I haven’t circled back to my original issue with the ML-provided search 
> buckets not being the right size but if I have time I’ll see if the issue was 
> failing to specify item-frequency at some point.
> 
> Cheers,
> 
> E.
> -
> On 8/23/17, 2:37 AM, "general-boun...@developer.marklogic.com on behalf of 
> Geert Josten"  geert.jos...@marklogic.com> wrote:
> 
>Hi Eliot,
> 
>Keep in mind that you pass in item-frequency in cts:element-values, but
>the default for range constraints is likely fragment-frequency. Did you
>pass in an item-frequency facet-option in there too?
> 
>Kind regards,
>Geert
> 
>On 8/22/17, 10:47 PM, "general-boun...@developer.marklogic.com on behalf
>of Eliot Kimber" ekim...@contrext.com> wrote:
> 
>> If I sum the counts of each bucket calculated using cts:frequency() it
>> matches the total calculated using the initial result from the
>> element-values() query, so I guess the 10,000 count is a side effect of
>> some internal lexicon implementation magic.
>> 
>> Cheers,
>> 
>> E.
>> 
>> --
>> Eliot Kimber
>> http://contrext.com
>> 
>> 
>> 
>> On 8/22/17, 3:25 PM, "general-boun...@developer.marklogic.com on behalf
>> of Eliot Kimber" > ekim...@contrext.com> wrote:
>> 
>>   I think this is again my weak understanding of lexicons and frequency
>> counting. 
>> 
>>   If I change my code to sum the frequencies of the durations in each
>> range then I get more sensible numbers, e.g.:
>> 
>>   let $count := sum(for $dur in $durations[. lt $upper-bound][. ge
>> $lower-bound] return cts:frequency($dur))
>> 
>>   Having updated get-enrichment-durations() to:
>> 
>>   cts:element-values(xs:QName("prof:overall-elapsed"), (),
>> ("descending", "item-frequency"),
>>cts:collection-query($collection))
>> 
>>   It still seems odd that the pure lexicon check returns exactly 10,000
>> *values*--that still seems suspect, but then using those 10,000 values to
>> calculate the total frequency does result in a more likely number. I
>> guess I can do some brute-force querying to see if it¹s accurate.
>> 
>>   Cheers,
>> 
>>   Eliot
>>   --
>>   Eliot Kimber
>>   http://contrext.com
>> 
>> 
>> 
>>   On 8/22/17, 2:52 PM, "general-boun...@developer.marklogic.com on
>> behalf of Eliot Kimber" > behalf of ekim...@contrext.com> wrote:
>> 
>>   Using ML 8.0-3.2
>> 
>>   As part of my profiling application I run a large number of
>> profiles, storing the profiler results back to the database. I¹m then
>> extracting the times from the profiling data to create histograms and do
>> other analysis.
>> 
>>   My first attempt to do this with buckets ran into the problem
>> that the index-based buckets were not returning accurate numbers, so I
>> reimplemented it to construct the buckets manually from a list of the
>> actual duration values.
>> 
>>   My code is:
>> 
>>   let $durations as xs:dayTimeDuration* :=
>> epf:get-enrichment-durations($collection)
>>   let $search-range := epf:construct-search-range()
>>   let $facets :=
>>   for $bucket in $search-range/search:bucket
>>   let $upper-bound := if ($bucket/@lt) then
>> xs:dayTimeDuration($bucket/@lt) else xs:dayTimeDuration("PT0S")
>>   let $lower-bound := xs:dayTimeDuration($bucket/@ge)
>>   let $count := count($durations[. lt $upper-bound][. ge
>> $lower-bound]) 
>>   return if ($count gt 0)
>>  then > count="{$count}">{epf:format-day-time-duration($upper-bound)}> t-value>
>>  else ()
>> 
>>   The get-enrichment-durations() function does this:
>> 
>> cts:element-values(xs:QName("prof:overall-elapsed"), (),
>> "descending",
>>cts:collection-query($collection))
>> 
>>   This works nicely and seems to provide 

Re: [MarkLogic Dev General] JSearch - collections

[Justin Makeig]

To extend on what Erik said, the collections() method is designed to provide a 
top-level filter for the set of documents you're interested in. It's generally 
a best practice to organize your documents into collections. For example, all 
documents representing customers might be in the `com.acme.customer` 
collection. For this purpose, collections are roughly equivalent to a table in 
a relational database: a means to segregate entities of the same type. In 
practice, collections are much more flexible than tables, though. 

In JavaScript you can then provide a shortcut such that all subsequent queries 
will be scoped to that particular collection, so you don't have to remember to 
add the cts.collectionQuery() each time:

const cust = jsearch.collections('com.acme.customers');
const prod = jsearch.collections('com.acme.products');

cust.documents().where(…).result();
prod.values(…).result();

If you were writing a library that operated over customers, you might even 
export the cust variable such that the consumer didn't even have to worry about 
the collection name itself.

If you're using collections for other purposes (and you should), you can 
combine a top-level collections() call with cts.colletionQuery() in the where() 
to get the intersection.

Justin

> On Aug 1, 2017, at 6:20 AM, Florent Georges  wrote:
> 
> Thank you, Erik!  I hoped there was a ready-to-use function for that.  Thank 
> you for confirming there is none.
> 
> I definitely owe you a drink of your choice after the past few days :-)
> 
> Regards,
> 
> -- 
> Florent Georges
> H2O Consulting
> http://h2o.consulting/
> 
> 
> On 1 August 2017 at 15:10, Erik Hennum wrote:
> Hi, Florent:
> 
> The collections() call is just a shortcut / convenience.
> 
> A full query can have a cts.andQuery() over a cts.collectionQuery() list.
> 
> 
> Hoping that helps,
> 
> 
> Erik Hennum
> 
> 
> From: general-boun...@developer.marklogic.com 
> [general-boun...@developer.marklogic.com] on behalf of Florent Georges 
> [li...@fgeorges.org]
> Sent: Tuesday, August 01, 2017 3:46 AM
> To: MarkLogic Developer Discussion
> Subject: [MarkLogic Dev General] JSearch - collections
> 
> Hi,
> 
> The jsearch.collections() function uses the union of the collections.  Is 
> there another function for the intersection instead?  That is, to select all 
> documents tagged with ALL the given collections (instead of ANY of them).
> 
> Regards,
> 
> -- 
> Florent Georges
> H2O Consulting
> http://h2o.consulting/ 
> 
> 
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
> 
> 
> 
> 
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Issue with Appservices after ML upgrade to ML9

[Justin Makeig]

Application Builder tried to simultaneously achieve two objectives

  1) quick, zero-code data browser and 
  2) application scaffolding for a real application. 

For a variety of reasons, it didn't do either particularly well and was thus 
caught in between. It had limited browsing capabilities and the code it 
generated, with its byzantine plugin architecture, was difficult to understand 
and extend—not the way you'd typically architect a "real" application. As a 
result, we decided to deprecate it in MarkLogic 8 and remove it in MarkLogic 9.

However, those two original use cases are definitely still on our radar. For 
the application scaffolding case, you should take a look at 
<https://github.com/marklogic-community/slush-marklogic-node>. The Slush 
generator <http://slushjs.github.io/> sets up a scaffolding for a three-tier 
Node application. There's also an ongoing initiative to implement the same 
capabilities in Java, namely Spring Boot 
<https://github.com/rjrudin/slush-marklogic-spring-boot>. Building on top of 
the Slush generated app, there's also an out-of-the-box discovery application 
<https://github.com/ryanjdew/ml-slush-discovery-app>. 

Longer term, we're also looking at various ways in the database to make it 
easier to figure out what data you have, how it's related, and how to assess 
its quality over time. 

Let me know if you have questions or concerns. Feedback from the community is 
always appreciated.

Justin


--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com

> On Jul 25, 2017, at 4:09 PM, Margare Cundiff <mcundi...@comcast.net> wrote:
> 
> Just curious - why was Application Builder junked? Is there anything in ML9 
> that is a replacement?
> 
> 
> 
> Thanks,
> 
> Morgan
> 
> 
> 
>> On July 23, 2017 at 12:44 PM Dave Cassel <david.cas...@marklogic.com> wrote:
>> 
>> /Appservices was deprecated in ML8 and removed in ML9. See the Release 
>> Notes: http://docs.marklogic.com/guide/relnotes/chap4#id_54314. 
>> 
>> -- 
>> Dave Cassel, @dmcassel
>> Technical Community Manager
>> MarkLogic Corporation
>> http://developer.marklogic.com/
>> 
>> From: <general-boun...@developer.marklogic.com> on behalf of amit gope 
>> <amitgo...@gmail.com>
>> Reply-To: MarkLogic Developer Discussion <general@developer.marklogic.com>
>> Date: Sunday, July 23, 2017 at 9:47 AM
>> To: "general@developer.marklogic.com" <general@developer.marklogic.com>
>> Subject: [MarkLogic Dev General] Issue with Appservices after ML upgrade to 
>> ML9
>> 
>> Hi All,
>> 
>> I am facing issue while trying to access the appservices URL 
>> http://localhost:8000/appservices. All other things working fine, please 
>> suggest what might be going wrong here. The Error that I am seeing is :
>> 
>> Error
>> An unexpected error has occurred. Check the error log or contact MarkLogic 
>> support.
>> 
>> 
>> Message
>> http://marklogic.com/xdmp/error error.xsd" 
>> xmlns:error="http://marklogic.com/xdmp/error; 
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
>> ">
>>   REST-INVALIDREQUEST
>>   
>>   1.0-ml
>>   Invalid request
>>   REST-INVALIDREQUEST: Invalid request: AppBuilder 
>> request no longer supported: /appservices
>>   false
>>   
>>   
>> AppBuilder request no longer supported: 
>> /appservices
>>   
>>   
>> 
>>   /MarkLogic/rest-api/8000-rewriter.xml
>>   1.0-ml
>> 
>>   
>> 
>> 
>>  
>> 
>> Best Regards
>> Amit 
>> 
> 
>  
> 
>> ___
>> General mailing list
>> General@developer.marklogic.com
>> Manage your subscription at: 
>> http://developer.marklogic.com/mailman/listinfo/general
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general






smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Convert output to PDF or Excel

[Justin Makeig]

> xdmp:set-response-content-type("application/pdf")


That only sets the Content-Type HTTP header. You'll still need to prepare the 
response body, i.e. the Excel or PDF itself. I wouldn't recommend trying to do 
that in XQuery. There are many libraries in Java, Node, Python, etc. that can 
take XML and turn it into one of those binary formats.

Justin

--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com

> On Jul 12, 2017, at 3:16 PM, Kari Cowan <kco...@alm.com> wrote:
> 
> Do you have an example with a bit more detail  Would you do it in xQuery or 
> would it need to be set in xcc/java response?
> 
> xdmp:set-response-content-type("application/pdf")
> xdmp:set-response-content-type("application/vnd.ms-excel")
> 
> On Mon, Jun 26, 2017 at 10:14 PM, VISHNU SINGH <svishnu.sin...@gmail.com> 
> wrote:
> We can use xdmp function as below:-
>   xdmp:set-response-content-type("application/vnd.ms-excel")
> 
> On 24 June 2017 at 01:42, Kari Cowan <kco...@alm.com> wrote:
> 
> Is there any function to convert an xqy query response to a PDF or Excel file?
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
> 
> 
> 
> 
> -- 
> Vishnu Singh | http://marklogicgd.blogspot.in/
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
> 
> 
> 
> 
> -- 
> Kari Cowan :: Web Developer :: ALM Media
> 1035 Market St. Suite 550, San Francisco, CA 94103
> iPhone/Text/FaceTime: +1 4157 444 762
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general




___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Search by age wise from dob property

[Justin Makeig]

const person = { dob: xs.date("1979-02-03") };
const thrirtyYearsAgo = fn.currentDate().subtract(xs.yearMonthDuration("P30Y"));
person.dob < thrirtyYearsAgo; // true

You can do date math with xs.duration types. In the above case, I'm subtracting 
30 years from the current date. xs.date.prototype.subtract() returns an 
xs.date. You can compare that xs.date to any other xs.date. To do this 
comparison in MarkLogic's indexes you'll need to create a range index 
. A range index, 
as its name implies, queries efficiently for ranges of typed values, for 
example, dates less than thirty ago from today.

cts.rangeQuery(cts.jsonPropertyReference('dob'), '<', thrirtyYearsAgo); // 
requires a range index of type xs:date on the dob JSON property


Justin


> On May 22, 2017, at 1:08 PM, Shiv Shankar  wrote:
> 
> Hi,
> There is a dob json property in the documents and I need to search on dob 
> based on age wise i.e age > 30, age >30 and age <50.  Any samples to 
> calculate age and compare in the search queries?
> 
> Thanks
> Shan.
> 
> 
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Is there a way to delete a stand-alone properties document?

[Justin Makeig]

First, why are you working with "stand-alone [un-sic.] properties documents"? 
Could your implementation be simplified by storing the properties in their own 
documents? There's no real advantage to a properties fragment, other than being 
able to access it with the same URI as its related document. Accessing the 
contents still requires a join between two fragments, just like with two 
separate documents. 

In your particular case, you should be able to remove the properties fragment 
with, xdmp:node-delete(xdmp:document-properties($uri)).

Justin

> On May 19, 2017, at 4:06 PM, Steve Anderson  wrote:
> 
> I'm doing some work with stand-along properties documents, and I want to be 
> able to delete them.  I know I can delete all the properties with a 
> 
> xdmp:document-set-properties($uri, ())
> 
> but that leaves an empty properties document hanging around.  Is there a way 
> to delete it?
> 
> Thanks,
> 
>   Steve
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] MarkLogic query console: Unable to log off/CSRF error

[Justin Makeig]

Query Console is not designed for simultaneous logins from the same user 
account. You should provision different users for individuals that need access 
to Query Console. With individual accounts you'll also be able to audit and 
revoke/reset on a user-by-user basis, general good security hygiene. 

The new Cross-Site Request Forgery 
<https://www.owasp.org/index.php/Cross-Site_Request_Forgery_(CSRF)> protections 
should give you a warning in the scenario you outline, but a page refresh 
should get a new token and restore your previous workspace. (If it doesn't, 
that's a bug. Please contact MarkLogic Support or me directly, off-list.) To 
logout in Chrome or Firefox you should quit your browser; it should prompt you 
to login again when you reopen.

If you had multiple user accounts accessing the same shared state you may be 
encountering a last-write-wins issue, where someone else zeroed out your 
workspace. Regardless, the best Query Console and security practice is to 
provision individual users. See 
<https://docs.marklogic.com/guide/qconsole/admin> and 
<https://docs.marklogic.com/guide/security/authentication>.

Justin


--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com


> On Apr 25, 2017, at 9:22 AM, Morrisey, Russell 
> <russell.morri...@cscglobal.com> wrote:
> 
> Hi all,
>  
> We are evaluating MarkLogic and we are running into a recurring problem with 
> the Query Console tool. We are using the evaluation version, v8.0-6.4.
>  
> When I visit the query console page, it automatically logs me in as the admin 
> user that we created during setup:
> mymarklogicserver:8000
> I want to log in as a different user, but there seems to be no log out button 
> in the Query Console interface.
> I have tried using Chrome Incognito mode, clearing my browser cache, and 
> switching to a different browser. I am stuck in a “logged in” state no matter 
> what I do.
>  
> If more than one of us uses the Query Console tool at the same time, I see a 
> popup telling me there is a CSRF error every time the other person does 
> something (like execute a query).
> This CSRF error pop-up can interrupt a query in mid-run.
> When I refresh after the CSRF error, everything that I did in the current 
> workspace (new query windows, query edits, etc.) is gone.
>  
> How can we fix this?
>  
> Thanks,
>  
> Russell Morrisey
> Sr. Java Developer | Shared Services Team
> russell.morri...@cscglobal.com
> cscglobal.com
>  
> 
>  
> Ask us about our new brand reveal – coming this May.
>  
> 
> 
> NOTICE: This e-mail and any attachments is intended only for use by the 
> addressee(s) named herein and may contain legally privileged, proprietary or 
> confidential information. If you are not the intended recipient of this 
> e-mail, you are hereby notified that any dissemination, distribution or 
> copying of this email, and any attachments thereto, is strictly prohibited. 
> If you receive this email in error please immediately notify me via reply 
> email or at (800) 927-9800 and permanently delete the original copy and any 
> copy of any e-mail, and any printout.
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general





smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] How to use uri-match in the java api?

[Justin Makeig]

Apologies, I read too fast. You were asking about "Java", not "JavaScript". 
(Car is to carpet as Java is to JavaScript.)

You'll need to specify a URI values constraint. 

http://marklogic.com/appservices/search;>
  

  
  …


And then queryManager.values("uris", new JacksonHandle()) to use it. (Hat tip: 
Sam Mefford.)

Take a look at <http://docs.marklogic.com/guide/java/searches#id_76144> and 
<http://docs.marklogic.com/guide/search-dev/appendixa#id_46397>.

Justin

> On Mar 31, 2017, at 8:08 AM, Justin Makeig <justin.mak...@marklogic.com> 
> wrote:
> 
> <https://docs.marklogic.com/cts.uriMatch>
> 
> In general, the built-in functions map one-to-one between JavaScript and 
> XQuery. (There are exceptions.) Because hyphens aren't friendly in JavaScript 
> property names, we do a kabab-case to camel-case translation for all of the 
> names in JavaScript, i.e. cts:uri-match() in XQuery is cts.uriMatch() in 
> JavaScript.
> 
> Justin
> 
>> On Mar 31, 2017, at 3:31 AM, Andreas Felix <andreas.fe...@ebcont.com> wrote:
>> 
>> Hi,
>> i need to fetch uris by wildcard, eg. foo*
>> In xquery i use the cts:uri-match function for this.
>> Does anybody know how so solve this with the Java-API?
>> 
>> Regards
>> Andreas
>> ___
>> General mailing list
>> General@developer.marklogic.com
>> Manage your subscription at: 
>> http://developer.marklogic.com/mailman/listinfo/general
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] How to use uri-match in the java api?

[Justin Makeig]

In general, the built-in functions map one-to-one between JavaScript and 
XQuery. (There are exceptions.) Because hyphens aren't friendly in JavaScript 
property names, we do a kabab-case to camel-case translation for all of the 
names in JavaScript, i.e. cts:uri-match() in XQuery is cts.uriMatch() in 
JavaScript.

Justin

> On Mar 31, 2017, at 3:31 AM, Andreas Felix  wrote:
> 
> Hi,
> i need to fetch uris by wildcard, eg. foo*
> In xquery i use the cts:uri-match function for this.
> Does anybody know how so solve this with the Java-API?
> 
> Regards
> Andreas
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] marklogic xquery to extract users and roles

[Justin Makeig]

For XQuery, take a look at the built-in functions in the `sec` (security) 
namespace <https://docs.marklogic.com/sec>.

If you're invoking this extract externally, you should also look at the 
Management API <https://docs.marklogic.com/REST/management/security>. This 
allows an authorized user to access the same configuration information over 
HTTP without having to write any custom code.

Justin


--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com


> On Mar 29, 2017, at 10:25 AM, Eric Shevchuk <eric.shevc...@bristowgroup.com> 
> wrote:
> 
> I am trying to create an xquery to extract users and roles does anyone have 
> such an xquery or can provide guidance on recommended best practices for 
> creating the xquery?
>  
> Thanks,
>  
> Eric Shevchuk
> Sr. Production Support/Applications Consultant
>  
> Infosys/Bristow Group, Inc.
> 2103 City West Blvd., 2nd Floor
> Houston, TX  77042
> eric.shevc...@bristowgroup.com
> 
>  
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general




___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Using RegEx in xQuery

[Justin Makeig]

The general way to recursively transform one node tree to another in XQuery is 
via recursive descent 
.
 

I can't quite tell from your description what your transformation needs to do, 
though. "Strip out everything in between" what? Can you give a more fleshed out 
example of the input and your expected output?

Justin



> On Mar 17, 2017, at 1:28 PM, Kari Cowan  wrote:
> 
> Inside my return for a query, I have an HTML node that I don’t need that 
> includes a bunch of child elements  -- I want to strip out everything in 
> between; alternately I could be happy to remove the node entirely from the 
> returned data.  Is there a function for that I don’t know about or is th is 
> better done with a regex?  Anyone have a tip they can share?
>  
> http://luxid.temis.com/occurrence/attribute; 
> xmlns:entityattr="http://luxid.temis.com/entity/attribute; 
> xmlns:entity="http://luxid.temis.com/entity; 
> xmlns:category="http://luxid.temis.com/category; xmlns="incisive-repository" 
> xmlns:i="incisive-repository">  (bunches of child elements here )  
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] MarkLogic temporality problem at insertion => Approach help

[Justin Makeig]

Are you sure your synthesized $generatedNodeNamspace is unique for each row? 
You might be trying to update the same URI more than once. To debug, you could 
replace your return statement with just `return $generatedNodeNamspace` and 
then count the distinct values. Does that equal $rowCount?

You could also simplify your FLWOR logic to:

for $row in fn:tokenize($rawDocument, ";")
let $columns := fn:tokenize($row, ",")
return …

If this is CSV-ish data you should also take care that values don't themselves 
have escaped commas or semicolons in them too.

Justin

> On Mar 14, 2017, at 9:39 AM, ZAKOWSKI Maximilien 
>  wrote:
> 
> I’ve got that code underneath, and I get the XDMP-CONFLICTINGUPDATES 
> Exception. In the exception description they are talking about semi-columns… 
> Do you know some example using that semi-column approach which can help me 
> inserting my temporal records?
>  
> https://help.marklogic.com/knowledgebase/article/View/25/16/xdmp-conflictingupdates-and-how-to-avoid-it
>  
> xquery version "1.0-ml";
> import module namespace temporal = "http://marklogic.com/xdmp/temporal; at 
> "/MarkLogic/temporal.xqy";
> declare namespace html = "http://www.w3.org/1999/xhtml;;
> let $rawDocument:= fn:doc("/docs/requete_perf_abs_20170307small.txt")
> let $rows := fn:tokenize($rawDocument, ";")
> let $rowCount := count($rows)
> for $currentRowIndex in 1 to $rowCount -1
>   let $currentRow := $rows[$currentRowIndex]
>   let $currentRowColumns := fn:tokenize($currentRow,",")
>   let $generatedNodeNamspace := 
> fn:concat($currentRowColumns[1],'-',$currentRowColumns[2],'-',$currentRowColumns[3],'-',$currentRowColumns[20],'.xml')
>   let $generatedNode :=
>   {$currentRowColumns[1]}
>   {$currentRowColumns[2]}
>   {$currentRowColumns[3]}
>   {$currentRowColumns[4]}
>   {$currentRowColumns[5]}
>   {$currentRowColumns[6]}
>   {$currentRowColumns[7]}
>   {$currentRowColumns[8]}
>   {$currentRowColumns[9]}
>   {$currentRowColumns[10]}
>   
> {$currentRowColumns[11]}
>   {$currentRowColumns[12]}
>   {$currentRowColumns[13]}
>   {$currentRowColumns[14]}
>   {$currentRowColumns[15]}
>   {$currentRowColumns[16]}
>   {$currentRowColumns[17]}
>   {$currentRowColumns[18]}
>   {$currentRowColumns[19]}
>   {$currentRowColumns[20]}
>   {$currentRowColumns[21]}
>   {$currentRowColumns[22]}
>   
>   
>   
>   
> 
>   return 
> temporal:document-insert("risk-metrics-temporality",$generatedNodeNamspace,$generatedNode);
> 
> 
> --
>  
> Ce message et toutes les pièces jointes sont confidentiels. Il est établi à 
> l'attention exclusive de son ou ses destinataire(s). Toute utilisation de ce 
> message non conforme à sa destination, toute diffusion ou toute publication, 
> totale ou partielle, est interdite, sauf autorisation expresse préalable. Son 
> contenu ne saurait constituer en aucun cas un engagement contractuel, une 
> offre de souscrire à quelconque produit ou instrument financier ou une 
> sollicitation à investir de la part de Le Groupe La Française et toutes 
> opinions exprimées dans ce message ne sauraient nécessairement refléter celle 
> de Le Groupe La Française. L'intégrité de ce message n'étant pas assurée sur 
> Internet, Le Groupe La Française ne saurait être responsable de son contenu. 
> Si vous recevez ce courriel par erreur, merci de le détruire et d'en avertir 
> immédiatement l'expéditeur. 
> Mentions légales disponibles sur le site Internet du Groupe à l’adresse 
> http://www.lafrancaise-group.com/mentions-legales.html 
> --
>  
> This email and any attachment are confidential and may be legally privileged 
> or otherwise protected from disclosure. It is intended only for the stated 
> addressee(s). Any use, dissemination or disclosure not in accordance with its 
> purpose, either in whole or in part, is prohibited without our prior formal 
> approval. Its contents, given solely for information, does not constitute a 
> commitment or an offer to subscribe to any financial product by Le Groupe La 
> Française any opinion expressed in this email may not necessarily reflect the 
> opinion of Le Groupe La Française. The integrity of this email cannot be 
> guaranteed through internet, Le Groupe La Française therefore is in no way 
> liable for any errors or omissions in the content of this message. If you are 
> neither the addressee nor an authorized recipient of this message, please 
> notify the sender of receipt and delete this message from 

Re: [MarkLogic Dev General] fn:doc() calls cached?

[Justin Makeig]

There are several levels of caching on most things that actually touch the data 
on disk. I'd encourage you to take a look at "Inside MarkLogic Server" 
<http://developer.marklogic.com/inside-marklogic>. It talks in detail about 
this and much more.

Justin

--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com

> On Dec 13, 2016, at 2:02 PM, Sekhon, Navdeep <navdeep.sek...@broadridge.com> 
> wrote:
> 
> Hi,
>  
> Are the calls to fn:doc() cached internally by marklogic or is the document 
> retrieved every time?
>  
> Regards,
>  
> Navdeep
>  
> This email transmitted on 100% recycled electrons.
>  
> 
> This message and any attachments are intended only for the use of the 
> addressee and may contain information that is privileged and confidential. If 
> the reader of the message is not the intended recipient or an authorized 
> representative of the intended recipient, you are hereby notified that any 
> dissemination of this communication is strictly prohibited. If you have 
> received this communication in error, please notify us immediately by e-mail 
> and delete the message and any attachments from your system.
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general






___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] MLCP export crashing with SEC-ROLEDNE

[Justin Makeig]

Oops. I just realized my example will only run in the upcoming MarkLogic 9, 
where we've updated the JavaScript engine to support ES2015, the latest version 
of the language. I'll leave it as an exercise for the reader to translate to 
MarkLogic 8-compatible ES5 or XQuery; the concepts are the same.

Happy coding!

Justin


> On Dec 10, 2016, at 11:37 AM, Justin Makeig <justin.mak...@marklogic.com> 
> wrote:
> 
> Because permissions are indexed, with the right incantation you can also 
> query by permissions. First you need to create a hash of the permission using 
> the role and capability, similar to the xdmp.permission constructor: 
> 
>  // JavaScript
>  xdmp.add64(
>xdmp.mul64(
>  xdmp.add64(
>xdmp.mul64(roleID, 5), xdmp.hash64(capability)
>  ), 5
>), xdmp.hash64('permission()')
>  );
> 
> Then use that hash to create a cts.termQuery (or in XQuery cts:term-query). 
> cts.termQuery is an undocumented query that matches the low-level terms that 
> make up the indexes.
> 
> You can use a cts.termQuery just like, or along with, any other cts.query. 
> For example,
> 
>  cts.uris(null, null, cts.termQuery(hash));
> 
> The example below allows you to specify one or more role IDs and it will 
> calculate the cross-product of roles and capabilities to build a query that 
> will match any document with any permission that uses those roles. (The 
> minimized blob at the top is just helper functions.)
> 
> Justin
> 
> 
> 
> 'use strict';
> 
> /** {@link 
> https://gist.github.com/jmakeig/0a331823ad9a458167f6#file-apply-as-9-sjs} */
> function applyAs(fct,options,thisArg){return function(){const 
> f=()=>{return[fct.call(thisArg,...arguments)]};options=options||{};if('string'===typeof
>  
> options.database){options.database=xdmp.database(options.database)}if(options.user){options.userId=xdmp.user(options.user);delete
>  
> options.user}if(fct.transactionMode&&!(options.transactionMode)){options.transactionMode=fct.transactionMode}return
>  fn.head(xdmp.invokeFunction(f,options)).pop();}};
> /** {@link 
> https://gist.github.com/jmakeig/0a331823ad9a458167f6#file-hof-mapper-sjs} */
> function map(fct,mapper){const ident=item=>item;mapper=mapper||ident;return 
> function*_map(){const itr=fct.apply(null,arguments);if('string'===typeof 
> itr||!(itr[Symbol.iterator])){yield itr}else{for(const item of itr){yield 
> mapper(item);
> 
> const sec = require('/MarkLogic/security');
> 
> /**
> * Generate hashes of permission terms for the cross product of roles and 
> capabilities (read, 
> * update, insert, execute). Use `cts.termQuery(hash)` to match documents by 
> their terms.
> *
> * @param   {string|Iterable.} [roleIDs] - One or more role IDs. 
> Defaults to all roles.
> * @returns {GeneratorFunction}  - A generator that yields 
> permission term hashes for the 
> * cross-product of roles and 
> capabilities
> */
> function* getPermissionTerms(roleIDs) {
>  const capabilities = ['read', 'update', 'insert', 'execute'];
>  /** @function */
>  const getRoleIDs = map(applyAs(sec.getRoleIds, { database: 
> xdmp.securityDatabase() }), fn.data);
>  function permHash(roleID, capability) {
>return xdmp.add64(
>  xdmp.mul64(
>xdmp.add64(
>  xdmp.mul64(roleID, 5), xdmp.hash64(capability)
>), 5
>  ), xdmp.hash64('permission()')
>);
>  }
>  for(const roleID of roleIDs || getRoleIDs()) {
>for(const capability of capabilities) {
>  yield permHash(roleID, capability);
>}
>  }
> }
> 
> const query = cts.orQuery(
>[...getPermissionTerms(/* Array of roleIDs here to limit to specific roles 
> */)]
>  .map(term => cts.termQuery(term))
> );
> 
> // URIs of the documents that have the supplied permissions
> cts.uris(null, null, query);
> 
> 
> 
> 
>> On Dec 8, 2016, at 6:15 PM, Will Thompson <wthomp...@oconnors.com> wrote:
>> 
>> I got a script working. I suspect there are faster/better ways to do this, 
>> but for anyone who may need it in the future, this will resolve any orphaned 
>> permissions. For a very large data set, you would probably need to batch 
>> this:
>> 
>> xquery version "1.0-ml";
>> 
>> let $uris := cts:uris((), 'limit=3')
>> let $permissions-map :=  map:new((
>> $uris ! map:entry(., xdmp:document-get-permissions(.))
>> ))
>> let $orphaned-map :=
>> xdmp:eval('
>> xquery version "1.0-ml";
>> import module namespace sec="http://marklogic.com/xdmp/security; at 
>>   "/MarkLogic/security.xqy";
&g

Re: [MarkLogic Dev General] MLCP export crashing with SEC-ROLEDNE

[Justin Makeig]

Because permissions are indexed, with the right incantation you can also query 
by permissions. First you need to create a hash of the permission using the 
role and capability, similar to the xdmp.permission constructor: 

  // JavaScript
  xdmp.add64(
xdmp.mul64(
  xdmp.add64(
xdmp.mul64(roleID, 5), xdmp.hash64(capability)
  ), 5
), xdmp.hash64('permission()')
  );

Then use that hash to create a cts.termQuery (or in XQuery cts:term-query). 
cts.termQuery is an undocumented query that matches the low-level terms that 
make up the indexes.

You can use a cts.termQuery just like, or along with, any other cts.query. For 
example,

  cts.uris(null, null, cts.termQuery(hash));

The example below allows you to specify one or more role IDs and it will 
calculate the cross-product of roles and capabilities to build a query that 
will match any document with any permission that uses those roles. (The 
minimized blob at the top is just helper functions.)

Justin



'use strict';

/** {@link 
https://gist.github.com/jmakeig/0a331823ad9a458167f6#file-apply-as-9-sjs} */
function applyAs(fct,options,thisArg){return function(){const 
f=()=>{return[fct.call(thisArg,...arguments)]};options=options||{};if('string'===typeof
 
options.database){options.database=xdmp.database(options.database)}if(options.user){options.userId=xdmp.user(options.user);delete
 
options.user}if(fct.transactionMode&&!(options.transactionMode)){options.transactionMode=fct.transactionMode}return
 fn.head(xdmp.invokeFunction(f,options)).pop();}};
/** {@link 
https://gist.github.com/jmakeig/0a331823ad9a458167f6#file-hof-mapper-sjs} */
function map(fct,mapper){const ident=item=>item;mapper=mapper||ident;return 
function*_map(){const itr=fct.apply(null,arguments);if('string'===typeof 
itr||!(itr[Symbol.iterator])){yield itr}else{for(const item of itr){yield 
mapper(item);

const sec = require('/MarkLogic/security');

/**
 * Generate hashes of permission terms for the cross product of roles and 
capabilities (read, 
 * update, insert, execute). Use `cts.termQuery(hash)` to match documents by 
their terms.
 *
 * @param   {string|Iterable.} [roleIDs] - One or more role IDs. 
Defaults to all roles.
 * @returns {GeneratorFunction}  - A generator that yields 
permission term hashes for the 
 * cross-product of roles and 
capabilities
 */
function* getPermissionTerms(roleIDs) {
  const capabilities = ['read', 'update', 'insert', 'execute'];
  /** @function */
  const getRoleIDs = map(applyAs(sec.getRoleIds, { database: 
xdmp.securityDatabase() }), fn.data);
  function permHash(roleID, capability) {
return xdmp.add64(
  xdmp.mul64(
xdmp.add64(
  xdmp.mul64(roleID, 5), xdmp.hash64(capability)
), 5
  ), xdmp.hash64('permission()')
);
  }
  for(const roleID of roleIDs || getRoleIDs()) {
for(const capability of capabilities) {
  yield permHash(roleID, capability);
}
  }
}

const query = cts.orQuery(
[...getPermissionTerms(/* Array of roleIDs here to limit to specific roles 
*/)]
  .map(term => cts.termQuery(term))
);

// URIs of the documents that have the supplied permissions
cts.uris(null, null, query);




> On Dec 8, 2016, at 6:15 PM, Will Thompson  wrote:
> 
> I got a script working. I suspect there are faster/better ways to do this, 
> but for anyone who may need it in the future, this will resolve any orphaned 
> permissions. For a very large data set, you would probably need to batch this:
> 
> xquery version "1.0-ml";
> 
> let $uris := cts:uris((), 'limit=3')
> let $permissions-map :=  map:new((
>  $uris ! map:entry(., xdmp:document-get-permissions(.))
>  ))
> let $orphaned-map :=
>  xdmp:eval('
>  xquery version "1.0-ml";
>  import module namespace sec="http://marklogic.com/xdmp/security; at 
>"/MarkLogic/security.xqy";
>   declare variable $PERMISSIONS external;
>   map:new(
> for $uri in map:keys($PERMISSIONS)
> let $orphaned := 
>   for $p in map:get($PERMISSIONS, $uri)
>   return try { 
> let $name := sec:get-role-names($p/sec:role-id) 
> return ()
>   }
>   catch ($e) { 
> if ($e/error:code = "SEC-ROLEDNE")
> then $p
> else xdmp:rethrow()
>   }
> where (exists($orphaned))
> return map:entry($uri, $orphaned)
>  )
>  ', 
>  (xs:QName('PERMISSIONS'), $permissions-map),
>  
>{xdmp:security-database()}
>  )
> for $o in map:keys($orphaned-map)
> let $permissions := map:get($orphaned-map, $o)
> return xdmp:document-remove-permissions($o, $permissions) 
> 
> -W
> 
>> On Dec 8, 2016, at 6:10 PM, Will Thompson  wrote:
>> 
>> Hi Chris,
>> 
>> That's entirely plausible and even likely. Do you know of an easy way to 
>> remove orphaned permissions? If there's not a shortcut, I could probably 
>> build a script to do it, but it seems like it might be messy: walk through 
>> every 

Re: [MarkLogic Dev General] MarkLogic 8.0-5 Indenting JSON output

[Justin Makeig]

Gary, I hope all is well with you.

Currently xdmp:output doesn't affect JSON nodes. It should. I'll add a Request 
For Enhancement (RFE) on my end to track this.

Here's a potential workaround:


(: See security guidance below :)
declare function json:stringify($json as node(), $spaces as xs:int) as 
xs:string? {
  xdmp:javascript-eval("JSON.stringify($json.toObject(), null, $spaces)", 
('$json', $json, '$spaces', $spaces))
};

declare function json:stringify($json as node()) as xs:string? {
  json:stringify($json, 2)
};

let $node := xdmp:unquote('["B", "B"]')
let $node :=  (: returns empty-sequence() :)
let $node := xdmp:unquote('{"a": "A", "b": ["B", "B"]}')
return json:stringify($node)

=> 

{
  "a": "A",
  "b": [
"B",
"B"
  ]
}

It uses xdmp:javascript-eval, which is potentially unsafe. You'd want to amp 
the json:stringify in a real production implementation so the eval can only be 
called in the context of json:stringify. See 
<https://docs.marklogic.com/guide/admin/security#id_81246>. (Note that 
JSON.stringify is *not* unsafe because it doesn't execute arbitrary 
JavaScript.) I haven't throughly tested the edge cases, but you get the idea.

Justin

--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com

> On Dec 1, 2016, at 9:55 AM, Gary Vidal <gvidalsass...@gmail.com> wrote:
> 
> All,
> 
> Is there an equivalent to format json output like
>  declare option xdmp:output "indent=yes";
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] finding the number of occurences of an element in an xml

[Justin Makeig]

The code below illustrates how you can calculate co-occurrences between an 
element and the URI of the documents that contain instances of that element. 
Then, for each URI it counts the total occurrences. Note, that you'll need to 
have the URI lexicon enabled and an element range index on x. 

Justin

(: Insert some dummy data :)
let $docs := (
  BBB,
  B,
  C,
  
)
return 
  for $doc at $i in $docs 
  return xdmp:document-insert($i || '.xml', $doc)
;
(: Calculate counts of  grouped by document URIs. Requires element range 
index on xs:QName('x') :)
let $co-occurr := cts:value-co-occurrences(cts:uri-reference(), 
cts:element-reference(xs:QName('x')), 'map')
for $uri in map:keys($co-occurr)
return $uri || ': ' || fn:count(map:get($co-occurr, $uri))



--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com


> On Nov 17, 2016, at 11:19 PM, Raghu <raghupathy.visweswa...@gmail.com> wrote:
> 
> Hi All,
> 
> I've got around 40 million XML documents out of which few documents are 
> having an element say element x twice (they are supposed to have only one 
> element x), I need to find the list of documents are there with multiple 
> occurrences of that element x. what would be the ideal way to query them?
> 
> Thanks in adavance
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general




___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Regarding Error in Marklogic Node

[Justin Makeig]

As the error message indicates, you should contact MarkLogic Support 
.

Justin

> On Nov 15, 2016, at 9:14 PM, Mani, Sivasubramani (ELS)  
> wrote:
> 
> Hi Team,
>  
> We have marklogic server with four nodes and We use first Node for ingestion 
> process using xdbc connector and that Node shows the following error
> “You've encountered an error in the server. If you have a maintenance 
> contract, you can open a support ticket by copying the text below and 
> emailing supp...@marklogic.com. Otherwise, please see our community Q & A 
> resources for help with this issue:
> 500: Internal Server Error
> XDMP-AS: (err:XPTY0004) $val as item() -- Invalid coercion: () as item()
> In /host-status.xqy on line 259
> In displayForm()
> $host = xs:unsignedLong("11572374834953542582")
> $snode = ()
> $sstats = ()
> $timenodes = ()
> In /host-status.xqy on line 946
> In hostStatusPage()
> In /host-status.xqy on line 963
> “ The application is in production and I am not able access it. Kindly do the 
> needful.
>  
> Thanks & Regards,
> Siva
>  
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Range indexes of document properties

[Justin Makeig]

> I am working with a database of text documents. I have extracted some 
> structured metadata from these documents and added these values as document 
> properties

I'd advise not using separate properties fragments for this. Joining properties 
and document fragments is not free.

The separation between document and properties fragments is nice and clean, but 
usually overkill. What you can do is co-locate the source text and its 
structured metadata in the same document:


  

  
  
   text
  


As you discover other interesting aspects of the raw input source, you can 
"promote" the structured, canonical forms into their own metadata elements 
without having to modify the source data. You can query either the metadata or 
the raw source or both. As before, you'd put a range index on the deliverydate 
element. However, if you have queries that cross metadata and source data, such 
as, "Find me all documents that mention the phrases X, Y, but not Z and were 
delivered last month", MarkLogic doesn't have to join separate fragments to 
resolve these.

"Envelope" is a common data modeling pattern in MarkLogic that takes advantage 
of the flexible document data model. Envelopes allow you to manage the various 
aspects of the same logical thing (e.g. customer, article, trade) in the same 
physical document. Because queries and caches generally work at the document 
level, this is also usually the most efficient way as well.

Obviously, there is lots more to data modeling in MarkLogic. I'd encourage 
those interested to check out the MarkLogic University courses on this topic 
<https://mlu.marklogic.com/ondemand/index.xqy?q=Series%3A%22Data%20Modeling%22>.

Justin

--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com

> On Nov 14, 2016, at 11:37 AM, Fox, David <d...@ap.org> wrote:
> 
> Thank you everyone for your responses. Geert was spot in with my misuse of 
> date, when I needed dateTime.
>  
> Also, great suggestions for how to combine searches across documents and 
> document properties. I like the suggestion to use cts: 
> properties-fragment-query together with search:resolve.
>  
> Thanks again,
> David
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] No option to pass query params in xdmp:http-get

[Justin Makeig]

You need to escape your ampersands if they appear in XQuery code.

  let $url as xs:string := "http://localhost:8080/q1=1q2=2q3=3;

Justin


> On Oct 31, 2016, at 9:01 AM, Shiv Shankar  wrote:
> 
> Hi,
> I tried calling an external ws with xdmp:http-get() which is deployed in 
> local host http://localhost:8080/q1=1=2=3 , it is throwing as
> XDMP-ENTITYREF: (err:XPST0003) Invalid entity reference "q2="
> 
> 
> I am trying to call external ws from console. Any advise?
> 
> Regards
> Shan.
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] General Digest, Vol 147, Issue 53

[Justin Makeig]

I still don't understand what your code is trying to do. Can you please 
describe in words what you're trying to accomplish? Where does `matViewUri` 
come from? How does it relate to `{personId: id}`?

Justin


> On Oct 3, 2016, at 7:08 AM, Shiv Shankar <shiv.shivshan...@gmail.com> wrote:
> 
> Hi Justin,
> Here is the actual code that returns  XDMP-CONFLICTINGUPDATES Error:. 
> 
> ---
> declareUpdate();
> var mainDataResponse =
>   rawColls.documents()
> .where(rawColls.byExample({personId: id}))
> .result('iterator'); 
> 
> for (var result of mainDataResponse.results) {
>
>   var normalizedViewObj = cts.doc(matViewUri).toObject();
>   
>   var matOriginalDoc = cts.doc(matViewUri);
> 
>   var dateNow =new Date().toUTCString();
>   
>  // if (doc.firstname !=null)
>   normalizedViewObj.firstName = result.document.firstname;
>   
>  // if (doc.lastName !=null)
> normalizedViewObj.lastName = result.document.lastname;
>  // if (doc.dateOfBirth !=null)
> normalizedViewObj.dateOfBirth = result.document.dob;
>  
>   xdmp.nodeReplace(cts.doc(matViewUri),normalizedViewObj),
>   normalizedViewObj = null;
> 
> }
> 
> ---
> 
> On Thu, Sep 29, 2016 at 3:00 PM, <general-requ...@developer.marklogic.com> 
> wrote:
> Send General mailing list submissions to
> general@developer.marklogic.com
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> http://developer.marklogic.com/mailman/listinfo/general
> or, via email, send a message with subject or body 'help' to
> general-requ...@developer.marklogic.com
> 
> You can reach the person managing the list at
> general-ow...@developer.marklogic.com
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of General digest..."
> 
> 
> Today's Topics:
> 
>1. Re: General Digest, Vol 147, Issue 51 (Justin Makeig)
> 
> 
> --
> 
> Message: 1
> Date: Thu, 29 Sep 2016 16:32:44 +
> From: Justin Makeig <justin.mak...@marklogic.com>
> Subject: Re: [MarkLogic Dev General] General Digest, Vol 147, Issue 51
> To: Shiv Shankar <shiv.shivshan...@gmail.com>
> Cc: "general@developer.marklogic.com"
> <general@developer.marklogic.com>
> Message-ID: <22fded82-fc67-4aad-b1f9-2eecb5b6a...@marklogic.com>
> Content-Type: text/plain; charset="us-ascii"
> 
> Can you show your actual code?
> 
> Justin
> 
> 
> --
> Justin Makeig
> Director, Product Management
> MarkLogic
> jmak...@marklogic.com
> 
> 
> > On Sep 29, 2016, at 7:09 AM, Shiv Shankar <shiv.shivshan...@gmail.com> 
> > wrote:
> >
> > HI Justin,
> > Thanks for your quick reply. However, I used both 
> > declareUpdate({explicitCommit: true});  and declareUpdate(); Still same 
> > issue.
> >
> > I am just reading from one collection and updating into another document in 
> > a different collection. I managed with insertDocument, but this over writes 
> > the docs which I don't want. I just want to update the attributes ( it is 
> > like UPDATE SET field1:=value, field2=value  in loop in oracle world)
> >
> > Note: I used lock/unlock options, but getting no lock on that target 
> > record. I used isolation level as part of eval. Couldn't get through it.
> > XDMP-CONFLICTINGUPDATES: xdmp.eval("//{explicitCommit: 
> > true}\ndeclareUpdate(); \nvar json = \"/Mark...", {},  > xmlns="xdmp:eval">67298589469423423423/<isolation...)
> >  --Conflicting updates xdmp.nodeReplace(cts.doc("
> >
> >
> > Any thoughts?
> >
> > On Thu, Sep 29, 2016 at 7:55 AM, <general-requ...@developer.marklogic.com> 
> > wrote:
> > Send General mailing list submissions to
> > general@developer.marklogic.com
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >     http://developer.marklogic.com/mailman/listinfo/general
> > or, via email, send a message with subject or body 'help' to
> > general-requ...@developer.marklogic.com
> >
> > You can reach the person managing the list at
> > general-ow...@developer.marklogic.com
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of General digest..."
> >
> >
> > Today's Topics:
> >
> >1. Re: XDMP-CONFLICTINGUPDATES: whileusing   xdmp.nodeReplace
> >   (Justin Makeig)
> >2. Re: mlcp Transaction Errors - SVC-EXTIME and 

Re: [MarkLogic Dev General] General Digest, Vol 147, Issue 51

[Justin Makeig]

Can you show your actual code? 

Justin


--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com


> On Sep 29, 2016, at 7:09 AM, Shiv Shankar <shiv.shivshan...@gmail.com> wrote:
> 
> HI Justin,
> Thanks for your quick reply. However, I used both 
> declareUpdate({explicitCommit: true});  and declareUpdate(); Still same issue.
> 
> I am just reading from one collection and updating into another document in a 
> different collection. I managed with insertDocument, but this over writes the 
> docs which I don't want. I just want to update the attributes ( it is like 
> UPDATE SET field1:=value, field2=value  in loop in oracle world)
> 
> Note: I used lock/unlock options, but getting no lock on that target record. 
> I used isolation level as part of eval. Couldn't get through it.
> XDMP-CONFLICTINGUPDATES: xdmp.eval("//{explicitCommit: 
> true}\ndeclareUpdate(); \nvar json = \"/Mark...", {},  xmlns="xdmp:eval">67298589469423423423/<isolation...)
>  --Conflicting updates xdmp.nodeReplace(cts.doc("
> 
> 
> Any thoughts?
> 
> On Thu, Sep 29, 2016 at 7:55 AM, <general-requ...@developer.marklogic.com> 
> wrote:
> Send General mailing list submissions to
> general@developer.marklogic.com
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> http://developer.marklogic.com/mailman/listinfo/general
> or, via email, send a message with subject or body 'help' to
> general-requ...@developer.marklogic.com
> 
> You can reach the person managing the list at
> general-ow...@developer.marklogic.com
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of General digest..."
> 
> 
> Today's Topics:
> 
>1. Re: XDMP-CONFLICTINGUPDATES: whileusing   xdmp.nodeReplace
>   (Justin Makeig)
>2. Re: mlcp Transaction Errors - SVC-EXTIME and XDMP-NOTXN #CGO#
>   (Jain, Abhishek)
> 
> 
> --
> 
> Message: 1
> Date: Wed, 28 Sep 2016 21:02:26 +
> From: Justin Makeig <justin.mak...@marklogic.com>
> Subject: Re: [MarkLogic Dev General] XDMP-CONFLICTINGUPDATES: while
> using   xdmp.nodeReplace
> To: MarkLogic Developer Discussion <general@developer.marklogic.com>
> Message-ID: <4136b8ca-113a-4373-b6d3-e54d34f15...@marklogic.com>
> Content-Type: text/plain; charset="us-ascii"
> 
> The general pattern for doing updates on persisted JSON documents in 
> JavaScript is to
>   1) get the document
>   2) turn it into an object
>   3) update the object
>   4) replace the document node with the updated object
> 
> Here's an example that updates the `balance` property of every document in 
> the `accounts` collection:
> 
> declareUpdate();
> for(var doc of fn.collection('accounts')) {
>   var account = doc.toObject();
>   account.balance = account.balance * 1.05;
>   xdmp.nodeReplace(doc, account);
> }
> 
> Note that you also need a declareUpdate(). The transaction is committed (or 
> rolledback) automatically, so you don't need the xdmp.commit().
> 
> Justin
> 
> --
> Justin Makeig
> Director, Product Management
> MarkLogic
> jmak...@marklogic.com
> 
> > On Sep 28, 2016, at 1:46 PM, Shiv Shankar <shiv.shivshan...@gmail.com> 
> > wrote:
> >
> > Hi Rob,
> > Thanks for quick reply,
> >
> > Here is the sample scenario.
> >
> > I would be building random data and put in to newObjData in a for loop.
> >
> > 01.json = {"firstName": "abc", "lastName": "xyz"}
> >
> > newObjectData.firstName= "Donald";
> >
> >  for Loop
> > {
> > xdmp.nodeReplace(cts.doc("/test/01.json"),newObjectData);
> > xdmp.commit();
> > }
> >
> >
> >
> >
> >
> > On Wed, Sep 28, 2016 at 4:37 PM, Rob Szkutak <rob.szku...@marklogic.com> 
> > wrote:
> > Hi,
> >
> > Do you have a reproducible case of this I could look at?
> >
> > Best,
> > Rob
> >
> > Rob Szkutak
> > Senior Consultant
> > MarkLogic Corporation
> > rob.szku...@marklogic.com
> > www.marklogic.com
> >
> > From: Shiv Shankar [shiv.shivshan...@gmail.com]
> > Sent: Wednesday, September 28, 2016 3:35 PM
> > To: general@developer.marklogic.com; Rob Szkutak
> > Subject: XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace
> >
> > Hi Rob,
> >
> > I am getting XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace to 
> > update JSON attributes  

Re: [MarkLogic Dev General] XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace

[Justin Makeig]

The general pattern for doing updates on persisted JSON documents in JavaScript 
is to 
  1) get the document 
  2) turn it into an object 
  3) update the object
  4) replace the document node with the updated object

Here's an example that updates the `balance` property of every document in the 
`accounts` collection: 

declareUpdate();
for(var doc of fn.collection('accounts')) {
  var account = doc.toObject();
  account.balance = account.balance * 1.05;
  xdmp.nodeReplace(doc, account);
} 

Note that you also need a declareUpdate(). The transaction is committed (or 
rolledback) automatically, so you don't need the xdmp.commit().

Justin

--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com

> On Sep 28, 2016, at 1:46 PM, Shiv Shankar <shiv.shivshan...@gmail.com> wrote:
> 
> Hi Rob,
> Thanks for quick reply,
> 
> Here is the sample scenario.
> 
> I would be building random data and put in to newObjData in a for loop.
> 
> 01.json = {"firstName": "abc", "lastName": "xyz"}
> 
> newObjectData.firstName= "Donald";
> 
>  for Loop
> {
> xdmp.nodeReplace(cts.doc("/test/01.json"),newObjectData);
> xdmp.commit();
> }
> 
> 
> 
> 
> 
> On Wed, Sep 28, 2016 at 4:37 PM, Rob Szkutak <rob.szku...@marklogic.com> 
> wrote:
> Hi,
> 
> Do you have a reproducible case of this I could look at?
> 
> Best,
> Rob
> 
> Rob Szkutak
> Senior Consultant
> MarkLogic Corporation
> rob.szku...@marklogic.com
> www.marklogic.com
> 
> From: Shiv Shankar [shiv.shivshan...@gmail.com]
> Sent: Wednesday, September 28, 2016 3:35 PM
> To: general@developer.marklogic.com; Rob Szkutak
> Subject: XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace
> 
> Hi Rob,
> 
> I am getting XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace to update 
> JSON attributes  in FOR loop, instead of over-writing entire document. Any 
> advice.
> 
> BTW, I used xdmp.eval, but it is not identifying the java script variables.
> 
> Thanks
> ShivShankar.
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Speeding up xquery returning aggregates

[Justin Makeig]

It’s the combinatorial explosion to get to those 38 tuples that’s the problem. 
What do the cardinalities of each of the “columns” (range indexes) look like? 
Is there a way you can reduce those?

Justin

--
Justin Makeig
Director, Product Management
MarkLogic


> On Sep 23, 2016, at 12:53 PM, Mark Shanks <markshanks...@hotmail.com> wrote:
> 
> I've already said it wasn't due to a high number of value-tuples. There are 
> only 38 value-tuples returned in total. Hence, limiting the result to the 
> first 100 [1 to 100] as you suggested is the same as the original query, and 
> the execution time is the same. I ran the code with your modification to 
> confirm this.
> From: general-boun...@developer.marklogic.com 
> <general-boun...@developer.marklogic.com> on behalf of Rob Szkutak 
> <rob.szku...@marklogic.com>
> Sent: Saturday, 24 September 2016 2:45:32 AM
> To: MarkLogic Developer Discussion
> Subject: Re: [MarkLogic Dev General] Speeding up xquery returning aggregates
>  
> Hi,
> 
> My assumption as I've written previously would be #3.
> 
> A very simple way to check would be cts:value-tuples()[100] . Adding the 
> [100] on the end would limit yourself to returning no more than the first 100 
> tuples of your result set. It wouldn't reduce the number of documents that 
> are evaluated. (To prove that, you could also do [100 to 200]). If your 
> theory about #2 is correct, then adding [100] shouldn't improve performance.
> 
> Best,
> Rob
> 
> Rob Szkutak
> Senior Consultant
> MarkLogic Corporation
> rob.szku...@marklogic.com
> www.marklogic.com
> 
> From: general-boun...@developer.marklogic.com 
> [general-boun...@developer.marklogic.com] on behalf of Mark Shanks 
> [markshanks...@hotmail.com]
> Sent: Friday, September 23, 2016 11:36 AM
> To: MarkLogic Developer Discussion
> Subject: Re: [MarkLogic Dev General] Speeding up xquery returning aggregates
> 
> Yes, many values were fine with a 10,000 document set but slowed down 
> massively when run against several million. To be clear, there are at least 3 
> counts we could be talking about. 1) The total number of documents in the 
> database. 2) The number of documents that the query is restricted to (such as 
> restricting to a certain date range). 3) The total number of value-tuples 
> returned. My experience is that the number 2 is driving the slowness (i.e., 
> the total number of value-tuples returned may be the same, but when marklogic 
> needs to determine this set over millions of documents rather than a small 
> number, performance degrades more than would be expected based on the number 
> alone, at least compared to the case of returning only 2 facets. 
> 
> I'm still unclear of what is going on under the hood in Marklogic. The 
> following link (https://docs.marklogic.com/guide/search-dev/lexicon) talks 
> about value co-occurrrence lexicons. If this is built, then 2 facets could 
> just refer to this and would result in the extremely fast performance 
> observed. On the other hand, 3 or more facets would not have a pre-prepared 
> lexicon to quiz. The documentation isn't clear whether a co-occurrence 
> lexicon is built whenever an index is built, or whether it needs to be 
> specifically configured. The documentation about creating lexicons points you 
> to the " 'Text Indexing' and 'Element/Attribute Range Indexes and Lexicons' 
> chapters of the Administrator's Guide", but these then don't mention 
> co-occurrence lexicons at all. So it isn't clear how you actually get a 
> co-occurrence lexicon built.
> 
> Thanks.
> Browsing With Lexicons (Search Developer's Guide ...
> docs.marklogic.com
> Browsing With Lexicons. MarkLogic Server allows you to create lexicons, which 
> are lists of unique words or values, either throughout an entire database 
> (words only ...
> 
> From: general-boun...@developer.marklogic.com 
> <general-boun...@developer.marklogic.com> on behalf of Rob Szkutak 
> <rob.szku...@marklogic.com>
> Sent: Friday, 23 September 2016 10:13:41 AM
> To: MarkLogic Developer Discussion
> Subject: Re: [MarkLogic Dev General] Speeding up xquery returning aggregates
>  
> Hi,
> 
> I thought in your earlier email you implied that many values were fine with a 
> 10,000 document set but slowed down when run against several million? This 
> lead me to believe the slowdown is caused by returning too many tuples.
> 
> A simple test to confirm if its a problem with the size of the result set 
> would be to limit the size of the result set and see if your performance 
> improves.
> 
> Best,
> Rob
> 
> Rob Szkutak
> Senior Consultant
> MarkLogic Corporation
> rob.szku...@marklogic.com
> www.marklogic.com
> 
&

Re: [MarkLogic Dev General] #CGO#How to ingest data of selected columns from CSV using MLCP & how to define and use primary key to see the log data (failed data in terms of Bad file)

[Justin Makeig]

Another benefit to staging and then transforming in two separate steps is that 
your transformation might require data that hasn't yet been loaded, for example 
to denormalize values from related documents that may or may not have already 
arrived.

Justin


--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com<mailto:jmak...@marklogic.com>

On Aug 31, 2016, at 7:06 AM, Dave Cassel 
<dave.cas...@marklogic.com<mailto:dave.cas...@marklogic.com>> wrote:

Hi Tim,

If you know at ingest time how you want to transform data, then from a 
performance point of view I think it's better to do it with the MLCP transform. 
Doing so means writing each fragment just once. There are some trade-offs:

Advantages of an mlcp transform:

  *   the data only need to be written once, instead of written and then 
updated. Doing the latter will result in deleted fragments, requiring merges to 
clean up.
  *   Once the data are in the database, they are fully ready for use — no need 
to segment freshly loaded data from data that is ready for use

Advantages of load-as-is followed by a CORB job:

  *   works if you don't know how you want to format the data — load, play, 
revise, repeat
  *   protection against errors in the transform: if your MLCP has a fatal 
error that affects only some documents, the whole batch will fail to get 
inserted. Good error handling can prevent this, but you may still need to 
account for not-fully-transformed documents.

Dave.

--
Dave Cassel<http://davidcassel.net/>, @dmcassel<https://twitter.com/dmcassel>
Technical Community Manager
MarkLogic Corporation<http://www.marklogic.com/>
http://developer.marklogic.com/


From: 
<general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>>
 on behalf of Timothy Taylor <timmy...@gmail.com<mailto:timmy...@gmail.com>>
Reply-To: MarkLogic Developer Discussion 
<general@developer.marklogic.com<mailto:general@developer.marklogic.com>>
Date: Monday, August 29, 2016 at 6:34 PM
To: MarkLogic Developer Discussion 
<general@developer.marklogic.com<mailto:general@developer.marklogic.com>>
Subject: Re: [MarkLogic Dev General] #CGO#How to ingest data of selected 
columns from CSV using MLCP & how to define and use primary key to see the log 
data (failed data in terms of Bad file)

Hey Dave,

Tim Taylor from the alliances team here. Subscribed from my personal email.

Any thoughts on whether an mlcp  transform on the inbound side versus load as 
is and run a CORB job to clean up afterwards would perform better?

Tim

Sent from my iPhone

On Aug 29, 2016, at 2:10 PM, Dave Cassel 
<dave.cas...@marklogic.com<mailto:dave.cas...@marklogic.com>> wrote:

You can write an MLCP 
transform<http://docs.marklogic.com/guide/mlcp/import#id_82518>. That should 
get the individual XML documents as input and your output can structure them 
however you want. This post on recursive 
descent<http://developer.marklogic.com/blog/xquery-recursive-descent> should 
help, too — you'll use that in your transform.

--
Dave Cassel<http://davidcassel.net/>, @dmcassel<https://twitter.com/dmcassel>
Technical Community Manager
MarkLogic Corporation<http://www.marklogic.com/>
http://developer.marklogic.com/


From: 
<general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>>
 on behalf of "Khan, Zishan" 
<zishan.k...@capgemini.com<mailto:zishan.k...@capgemini.com>>
Reply-To: MarkLogic Developer Discussion 
<general@developer.marklogic.com<mailto:general@developer.marklogic.com>>
Date: Monday, August 29, 2016 at 8:12 AM
To: MarkLogic Developer Discussion 
<general@developer.marklogic.com<mailto:general@developer.marklogic.com>>
Subject: [MarkLogic Dev General] #CGO#How to ingest data of selected columns 
from CSV using MLCP & how to define and use primary key to see the log data 
(failed data in terms of Bad file)

Hi Folk,

Being a newbie in MarkLogic, I need your serious response to come out of faced 
challenges.
The moment when I was ingesting CSV Structured data into MarkLogic as-is to get 
XML output data in our database using MLCP, I am ok with normal ingestion using 
MLCP in any file format but I got stuck in finding the solution of below 
mentioned problem

1.  How to ingest data corresponding to selected columns only in MarkLogic 
using MLCP or by any means.
2.  How to define and use Primary key to check the logs ( say failed data ).

For the ease of understanding I am exploring my doubts with below example.

Input :

idemp_name  salarydesignation mobile_no 
  dependent

1 ABC   3000  X
442
2 DEF   4000  Y
221  

Re: [MarkLogic Dev General] xdmp.getRequestField return type

[Justin Makeig]

If you look at the XQuery version 
<https://docs.marklogic.com/xdmp:get-request-field>, it returns item()*. This 
accounts for the cases where you've got strings zero, one, or many strings in 
application/x-www-form-urlencoded or binary in multipart/form-data.

We're looking at ways to be more precise with how we document JavaScript types.

Justin

--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com<mailto:jmak...@marklogic.com>

On Aug 26, 2016, at 2:23 AM, Florent Georges 
<li...@fgeorges.org<mailto:li...@fgeorges.org>> wrote:

Hi,

https://docs.marklogic.com/xdmp.getRequestField tells me that the function 
returns "Object?".  But "typeof" on the result of calling it (with a simple 
"param=value" in the URL) tells me "string".

Is that a bug in the documentation?  Or does the notation "Object?" allows for 
a plain string as well?

Regards,

--
Florent Georges
http://fgeorges.org/
http://h2oconsulting.be/

___
General mailing list
General@developer.marklogic.com<mailto:General@developer.marklogic.com>
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] How to merge two json document in to another document in MarkLogic

[Justin Makeig]

What does is the merged document supposed to look like? What is the logic when 
key names collide? Last wins?

Justin

> On Aug 16, 2016, at 1:34 PM, Siva Viswabrahmana - Contractor 
> <siva.viswabrahm...@scdhhs.gov> wrote:
> 
> Hi,
> 
> /1.json:
> {“id”:”1”, “fname”:”Scott”}
> 
> /2.json:
> {“id”:”2”, “lname”:”McNelay”}
> 
> Hope this works.
> 
> —Siva.
> 
> From: <general-boun...@developer.marklogic.com 
> <mailto:general-boun...@developer.marklogic.com>> on behalf of Justin Makeig 
> <justin.mak...@marklogic.com <mailto:justin.mak...@marklogic.com>>
> Reply-To: MarkLogic Developer Discussion <general@developer.marklogic.com 
> <mailto:general@developer.marklogic.com>>
> Date: Tuesday, August 16, 2016 at 4:31 PM
> To: MarkLogic Developer Discussion <general@developer.marklogic.com 
> <mailto:general@developer.marklogic.com>>
> Subject: Re: [MarkLogic Dev General] How to merge two json document in to 
> another document in MarkLogic
> 
> Can you show an examples of 1.json, 2.json, and 12.json? It's unclear what 
> you mean by merge. 
> 
> Justin
> 
> --
> Justin Makeig
> Director, Product Management
> MarkLogic
> justin.mak...@marklogic.com <mailto:justin.mak...@marklogic.com>
> 
>> On Aug 16, 2016, at 1:19 PM, Siva Viswabrahmana - Contractor 
>> <siva.viswabrahm...@scdhhs.gov <mailto:siva.viswabrahm...@scdhhs.gov>> wrote:
>> 
>> Hi,
>> I am trying to merge /1.json +/2.json into /12.json
>> 
>> Below query does not allow another document{…}
>> 
>> xdmp:document-insert("/test/2.json",document { doc("/test/1.json") 
>> },(),("aggregate")),
>> doc("/test/2.json”);
>> 
>> Pls advise.
>> 
>> Regards
>> Siva.
>> 
>> From: <general-boun...@developer.marklogic.com 
>> <mailto:general-boun...@developer.marklogic.com>> on behalf of Rob Szkutak 
>> <rob.szku...@marklogic.com <mailto:rob.szku...@marklogic.com>>
>> Reply-To: MarkLogic Developer Discussion <general@developer.marklogic.com 
>> <mailto:general@developer.marklogic.com>>
>> Date: Tuesday, August 16, 2016 at 4:16 PM
>> To: MarkLogic Developer Discussion <general@developer.marklogic.com 
>> <mailto:general@developer.marklogic.com>>
>> Subject: Re: [MarkLogic Dev General] How to merge two json document in to 
>> another document in MarkLogic
>> 
>> Hello,
>> 
>> Could you be more specific about what you're trying to accomplish? A sample 
>> of the structure of your two documents and the merged format you would like 
>> to put them in would be helpful in understanding how to assist you.
>> 
>> Best,
>> Rob
>> 
>> Rob Szkutak
>> Senior Consultant
>> MarkLogic Corporation
>> rob.szku...@marklogic.com <mailto:rob.szku...@marklogic.com>
>> www.marklogic.com 
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marklogic.com=DQMFAw=l2yuVHfpC_9lAv0gltv6ZQ=cN7CeyOq0lJM02AripOo2ttwg5D3eXuv33TcKhL4pCc=rNydfvxVonydW6JhTkm2obiz9yoCm1Xc6bz9NItJESQ=upWKrIYFG07ljVGDMxSd98ACdlKEv9wQruWfiO4l7t0=>
>> 
>> From: general-boun...@developer.marklogic.com 
>> <mailto:general-boun...@developer.marklogic.com> 
>> [general-boun...@developer.marklogic.com 
>> <mailto:general-boun...@developer.marklogic.com>] on behalf of Siva 
>> Viswabrahmana - Contractor [siva.viswabrahm...@scdhhs.gov 
>> <mailto:siva.viswabrahm...@scdhhs.gov>]
>> Sent: Tuesday, August 16, 2016 3:11 PM
>> To: general@developer.marklogic.com <mailto:general@developer.marklogic.com>
>> Subject: [MarkLogic Dev General] How to merge two json document in to 
>> another document in MarkLogic
>> 
>> Hi,
>> How to merge two json document in to another document in MarkLogic?
>> 
>> Regards
>> Siva.
>> Confidentiality Note 
>> 
>> 
>> This message is intended for the use of the person or entity to which it is 
>> addressed and may contain information, including health information, that is 
>> privileged, confidential, and the disclosure of which is governed by 
>> applicable law. If the reader of this message is not the intended recipient, 
>> or the employee or agent responsible to deliver it to the intended 
>> recipient, you are hereby notified that any dissemination, distribution or 
>> copying of this information is STRICTLY PROHIBITED. 
>> 
>> If you have received this in error, please notify us immediately and destroy 
>> the related message.
>> Confidentiality Note 
>> 
>

Re: [MarkLogic Dev General] How to merge two json document in to another document in MarkLogic

[Justin Makeig]

Can you show an examples of 1.json, 2.json, and 12.json? It's unclear what you 
mean by merge. 

Justin

--
Justin Makeig
Director, Product Management
MarkLogic
justin.mak...@marklogic.com

> On Aug 16, 2016, at 1:19 PM, Siva Viswabrahmana - Contractor 
> <siva.viswabrahm...@scdhhs.gov> wrote:
> 
> Hi,
> I am trying to merge /1.json +/2.json into /12.json
> 
> Below query does not allow another document{…}
> 
> xdmp:document-insert("/test/2.json",document { doc("/test/1.json") 
> },(),("aggregate")),
> doc("/test/2.json”);
> 
> Pls advise.
> 
> Regards
> Siva.
> 
> From: <general-boun...@developer.marklogic.com 
> <mailto:general-boun...@developer.marklogic.com>> on behalf of Rob Szkutak 
> <rob.szku...@marklogic.com <mailto:rob.szku...@marklogic.com>>
> Reply-To: MarkLogic Developer Discussion <general@developer.marklogic.com 
> <mailto:general@developer.marklogic.com>>
> Date: Tuesday, August 16, 2016 at 4:16 PM
> To: MarkLogic Developer Discussion <general@developer.marklogic.com 
> <mailto:general@developer.marklogic.com>>
> Subject: Re: [MarkLogic Dev General] How to merge two json document in to 
> another document in MarkLogic
> 
> Hello,
> 
> Could you be more specific about what you're trying to accomplish? A sample 
> of the structure of your two documents and the merged format you would like 
> to put them in would be helpful in understanding how to assist you.
> 
> Best,
> Rob
> 
> Rob Szkutak
> Senior Consultant
> MarkLogic Corporation
> rob.szku...@marklogic.com <mailto:rob.szku...@marklogic.com>
> www.marklogic.com 
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marklogic.com=DQMFAw=l2yuVHfpC_9lAv0gltv6ZQ=cN7CeyOq0lJM02AripOo2ttwg5D3eXuv33TcKhL4pCc=rNydfvxVonydW6JhTkm2obiz9yoCm1Xc6bz9NItJESQ=upWKrIYFG07ljVGDMxSd98ACdlKEv9wQruWfiO4l7t0=>
> 
> From: general-boun...@developer.marklogic.com 
> <mailto:general-boun...@developer.marklogic.com> 
> [general-boun...@developer.marklogic.com 
> <mailto:general-boun...@developer.marklogic.com>] on behalf of Siva 
> Viswabrahmana - Contractor [siva.viswabrahm...@scdhhs.gov 
> <mailto:siva.viswabrahm...@scdhhs.gov>]
> Sent: Tuesday, August 16, 2016 3:11 PM
> To: general@developer.marklogic.com <mailto:general@developer.marklogic.com>
> Subject: [MarkLogic Dev General] How to merge two json document in to another 
> document in MarkLogic
> 
> Hi,
> How to merge two json document in to another document in MarkLogic?
> 
> Regards
> Siva.
> Confidentiality Note 
> 
> 
> This message is intended for the use of the person or entity to which it is 
> addressed and may contain information, including health information, that is 
> privileged, confidential, and the disclosure of which is governed by 
> applicable law. If the reader of this message is not the intended recipient, 
> or the employee or agent responsible to deliver it to the intended recipient, 
> you are hereby notified that any dissemination, distribution or copying of 
> this information is STRICTLY PROHIBITED. 
> 
> If you have received this in error, please notify us immediately and destroy 
> the related message.
> Confidentiality Note 
> 
> 
> This message is intended for the use of the person or entity to which it is 
> addressed and may contain information, including health information, that is 
> privileged, confidential, and the disclosure of which is governed by 
> applicable law. If the reader of this message is not the intended recipient, 
> or the employee or agent responsible to deliver it to the intended recipient, 
> you are hereby notified that any dissemination, distribution or copying of 
> this information is STRICTLY PROHIBITED. 
> 
> If you have received this in error, please notify us immediately and destroy 
> the related message. ___
> General mailing list
> General@developer.marklogic.com <mailto:General@developer.marklogic.com>
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general 
> <http://developer.marklogic.com/mailman/listinfo/general>



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] XDMP-NOTXN | Issue with MLCP while hitting load-balancer

[Justin Makeig]

A transaction is scoped only to the host that initiated it. You'll need to 
configure your load balancer to maintain host affinity for transactions. Better 
yet, don't use a load balancer at all with mlcp. mlcp uses its own logic to 
communicate with multiple hosts in a cluster. Because it knows the details of 
the cluster topology, mlcp can route requests better than a generic HTTP load 
balancer will.

Justin

--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com

> On Aug 11, 2016, at 1:49 AM, Rashmi Ranjan Acharya <itsonlyras...@gmail.com> 
> wrote:
> 
> Hi Folks, 
> During content ingestion, MLCP is losing some transactions while hitting 
> load-balancer with the below exception but it works completely fine when it 
> hits a particular host/node.
> 
>  
> 
> Exception snippet:  ERROR mapreduce.ContentWriter: 
> com.marklogic.xcc.exceptions.XQueryException: XDMP-NOTXN: No transaction with 
> identifier 115728575…
> We are using Mark logic server 8.0-5.5 with XCC client 8.0-5; and want to use 
> load-balancer without losing any transaction.
> 
> Can anyone suggest any solution to overcome this?
> 
> Thanks in advance !
> 
> 
> 
> Regards,
> 
> Rashmi Ranjan Acharya
> 
> 
> 
> 
> -- 
> Rashmi Ranjan Acharya
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Bulk updates (xqsync vs. mlcp)

[Justin Makeig]

Just to wrap up this thread, I've incorrectly and unhelpfully conflated two 
aspects of merging: consolidating stands and getting rid of obsolete fragments. 
When you set a merge timestamp you are only affecting the latter. Regardless of 
the timestamp the database will always consolidate stands for you. Again, the 
docs have very good coverage of this 
<https://docs.marklogic.com/guide/admin/merges>. (Hat tip, Jason Hunter and 
Danny Sokolsky.)

Regardless, I still stand by the recommendation to _not_ use MVCC timestamps as 
general-purpose versioning, mostly for the difficulty in querying and the 
potential to screw something up administratively.

Sorry for the confusion. 

Justin

--
Justin Makeig
Director, Product Management
MarkLogic
justin.mak...@marklogic.com

> On Jun 30, 2016, at 11:02 AM, Hans Hübner <hans.hueb...@lambdawerk.com> wrote:
> 
> On Thu, Jun 30, 2016 at 6:30 PM, Justin Makeig <justin.mak...@marklogic.com 
> <mailto:justin.mak...@marklogic.com>> wrote:
>> The amount of data that we're accumulating by keeping the old versions 
>> around does not bother us.  
> 
> It will bother you if you never allow the database to merge. If you're 
> keeping a small window of history this will work fine. (Though an errant 
> timestamp setting by a config script will delete your history.) If you need 
> to keep the entire history, you will effectively be disabling merging with 
> this strategy, which will certainly land you in trouble. Merges are good; the 
> database needs them to optimize its internal data structures to support fast 
> and consistent ingest and queries.
> 
> Save each version as a separate document. Put all versions of a single 
> document in a collection to represent the "logical" document and give each 
> instance version a unique URI to represent its version number. You could even 
> create a special "latest" collection that contains only the latest version of 
> each document. This will allow you to do queries like, "How many versions of 
> (logical) document ABC.xml do I have?" "What's the latest version of 
> (logical) ABC.xml?" "Run this diff code on latest ABC.xml and its previous 
> version." With timestamps you'll have to know _when_ a document was updated 
> in order to get its previous version. This will require two steps for every 
> query and won't allow you to do any queries across versions, because the 
> older/newer versions don't exist, from the perspective of a query that runs 
> at a single timestamp.  
> 
> Thank you for the concrete architectural advice!  It does not seem to be very 
> bothersome to follow that route, so we will certainly trust you in that it is 
> better than using MVCC timestamps.
> 
> Let me suggest again that the "Time Travel" section in the "Inside Marklogic" 
> document and the section on point-in-time queries in the "Application 
> Developers Guide" be updated to include information on the caveats that you 
> and your colleagues have expressed.  I'm still a bit puzzled by the vehemence 
> that you all put forth into discouraging us from using it.  Are there any 
> other advertised features that can affect the health of a database in a 
> similar way and should thus be avoided?
> 
> Thanks!
> Hans
> 
> -- 
> LambdaWerk GmbH
> Oranienburger Straße 87/89
> 10178 Berlin
> Phone: +49 30 555 7335 0
> Fax: +49 30 555 7335 99
> 
> HRB 169991 B Amtsgericht Charlottenburg
> USt-ID: DE301399951
> Geschäftsführer:  Hans Hübner
> 
> http://lambdawerk.com/ <http://lambdawerk.com/>
> 
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Bulk updates (xqsync vs. mlcp)

[Justin Makeig]

The amount of data that we're accumulating by keeping the old versions around 
does not bother us.

It will bother you if you never allow the database to merge. If you're keeping 
a small window of history this will work fine. (Though an errant timestamp 
setting by a config script will delete your history.) If you need to keep the 
entire history, you will effectively be disabling merging with this strategy, 
which will certainly land you in trouble. Merges are good; the database needs 
them to optimize its internal data structures to support fast and consistent 
ingest and queries.

Save each version as a separate document. Put all versions of a single document 
in a collection to represent the "logical" document and give each instance 
version a unique URI to represent its version number. You could even create a 
special "latest" collection that contains only the latest version of each 
document. This will allow you to do queries like, "How many versions of 
(logical) document ABC.xml do I have?" "What's the latest version of (logical) 
ABC.xml?" "Run this diff code on latest ABC.xml and its previous version." With 
timestamps you'll have to know _when_ a document was updated in order to get 
its previous version. This will require two steps for every query and won't 
allow you to do any queries across versions, because the older/newer versions 
don't exist, from the perspective of a query that runs at a single timestamp.

Justin

On Jun 29, 2016, at 8:49 PM, Hans Hübner 
> wrote:

On Wed, Jun 29, 2016 at 11:29 PM, Danny Sokolsky 
> wrote:
It might be tempting to treat point-in-time queries for generic versioning, but 
it is usually not what you want.

Does that help to clarify?

Thanks, Danny, this helps.  In our use case, we have thousands of relatively 
complex trees of nodes, and the configuration of each tree changes over time, 
when new data is inserted into the database.  In order to make old 
configurations of each tree available for inspection, we use the MVCC 
point-in-time rollback feature of our current database system to recover 
previous database states and visualize them.  This is merely a diagnostic 
feature, but given the relative complexity of the connections between the tree 
nodes, it is helpful to be able to visualize the changes to each tree that 
happened when new data was inserted.

The amount of data that we're accumulating by keeping the old versions around 
does not bother us.  This database is a special purpose database tied to a 
particular application, and it won't be used to insert random other documents.  
It thus seems to me that we'll be fine with using the MVCC feature for our 
history visualization for now.  If we decide that the space overhead is 
prohibitive, we can always adjust the merge timestamp, trading off history 
depth against database space used.

It would be helpful to have the tradeoffs that one has to make when using the 
"Time Travel" feature be listed in the documentation.

-Hans

--
LambdaWerk GmbH
Oranienburger Straße 87/89
10178 Berlin
Phone: +49 30 555 7335 0
Fax: +49 30 555 7335 99

HRB 169991 B Amtsgericht Charlottenburg
USt-ID: DE301399951
Geschäftsführer:  Hans Hübner

http://lambdawerk.com/


___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Bulk updates (xqsync vs. mlcp)

[Justin Makeig]

 the point-in-time feature does not require that we disable merges.  It just 
requires that the merge timestamp is set to the earliest point back in time 
where we want to be able to look back to.

Yes. That's correct. The further you push the merge timestamp back, the more 
your going to stress normal operations, though. Merges allow the database to 
optimize the storage and indexes to support high-performance I/O. They're not a 
nice-to-have, they're a required aspect of how MarkLogic works. The fact that 
you can delay or turn them off is an advanced operation for special cases, such 
as rolling back a database for disaster recovery.

What I am still missing is why the "Inside MarkLogic" document describes how 
MVCC timestamps can be used to implement "Time Travel" and the "Application 
Developer's Guide" describe point-in-time queries if you (assuming that you 
speak for MarkLogic) advise against using them.

Point-in-time queries are good for "micro" time travel, if I may coin a term. 
They're good for maintaining a consistent snapshot over a very short period of 
time, within the finite window that you'd configure to not merge. Beyond that 
window, the history is gone—optimized away. If you need to keep that history 
you should do so explicitly in documents. (That's how the Document Library 
Services and Bitemporal APIs maintain version histories.)

Is the documentation accurate?

Yes, but a little light on why you'd use point-in-time queries and what the 
boundaries and implications are.

Under what circumstances do you recommend using the point-in-time technique 
described in the guide?  Does the point-in-time query technique only work if 
merges are disabled?

Yes, doing anything at a point in time in the past means that you need to 
maintain the (MVCC) state back to that point. The only way the database can do 
that is by not merging out all of the obsolete fragments. This is OK for finite 
windows of time, but the database needs to eventually merge. (You can give the 
merge timestamp a negative value to maintain a rolling window.)


Justin


--
Justin Makeig
Director, Product Management
MarkLogic
justin.mak...@marklogic.com<mailto:justin.mak...@marklogic.com>

On Jun 29, 2016, at 12:18 PM, Hans Hübner 
<hans.hueb...@lambdawerk.com<mailto:hans.hueb...@lambdawerk.com>> wrote:

Justin,

thank you for the additional documentation pointer.  From what I read, I 
understand that merging is a useful operation and that merges should not be 
disabled.  I can agree to that, but as far as I have understood, the 
point-in-time feature does not require that we disable merges.  It just 
requires that the merge timestamp is set to the earliest point back in time 
where we want to be able to look back to.  Does setting the merge timestamp 
automatically disable the merges?

What I am still missing is why the "Inside MarkLogic" document describes how 
MVCC timestamps can be used to implement "Time Travel" and the "Application 
Developer's Guide" describe point-in-time queries if you (assuming that you 
speak for MarkLogic) advise against using them.  The "Application Developer's 
Guide" in particular describes how such queries work, in detail, and it does 
not mention that one should avoid the technique.

Is the documentation accurate?  Under what circumstances do you recommend using 
the point-in-time technique described in the guide?  Does the point-in-time 
query technique only work if merges are disabled?

Hans

On Wed, Jun 29, 2016 at 7:40 PM, Justin Makeig 
<justin.mak...@marklogic.com<mailto:justin.mak...@marklogic.com>> wrote:
Can you elaborate what you mean by "maintain the health of a database"?  If 
we'd decide that we never want to delete any data in a certain MarkLogic 
database so that we can roll back to any point in time, what would be the down 
sides?  How would the database become unhealthy?

Please take a look at the docs on merging, specifically the section, "Merges 
Are Good" <https://docs.marklogic.com/guide/admin/merges#id_43904>. Merging is 
the way that MarkLogic manages its internal data to support efficient and 
consistent ingest and query I/O. It is an internal process and completely 
orthogonal to how you version your documents.

What you describe sounds more like temporal versioning. Please take a look at 
MarkLogic's bitemporal APIs <https://docs.marklogic.com/guide/temporal/intro>. 
With bitemporal management you maintain an immutable copy of the entire history 
of your data that you can query at any point in time. The APIs do all of the 
sophisticated work maintaining versions securely. The "bi" in bitemporal allows 
you to query the valid time of the document (e.g. a trade was effective on 
2016-06-01) as you knew it at any point in time (e.g. the trade wasn't recorded 
until 2016-06-02 and then it was corrected on 2016-06-05).

Re: [MarkLogic Dev General] Bulk updates (xqsync vs. mlcp)

[Justin Makeig]

Can you elaborate what you mean by "maintain the health of a database"?  If 
we'd decide that we never want to delete any data in a certain MarkLogic 
database so that we can roll back to any point in time, what would be the down 
sides?  How would the database become unhealthy?

Please take a look at the docs on merging, specifically the section, "Merges 
Are Good" <https://docs.marklogic.com/guide/admin/merges#id_43904>. Merging is 
the way that MarkLogic manages its internal data to support efficient and 
consistent ingest and query I/O. It is an internal process and completely 
orthogonal to how you version your documents.

What you describe sounds more like temporal versioning. Please take a look at 
MarkLogic's bitemporal APIs <https://docs.marklogic.com/guide/temporal/intro>. 
With bitemporal management you maintain an immutable copy of the entire history 
of your data that you can query at any point in time. The APIs do all of the 
sophisticated work maintaining versions securely. The "bi" in bitemporal allows 
you to query the valid time of the document (e.g. a trade was effective on 
2016-06-01) as you knew it at any point in time (e.g. the trade wasn't recorded 
until 2016-06-02 and then it was corrected on 2016-06-05).

Justin


On Jun 28, 2016, at 9:55 PM, Hans Hübner 
<hans.hueb...@lambdawerk.com<mailto:hans.hueb...@lambdawerk.com>> wrote:

On Tue, Jun 28, 2016 at 10:36 PM, Justin Makeig 
<justin.mak...@marklogic.com<mailto:justin.mak...@marklogic.com>> wrote:
> as we want to be able to use the point-in-time query feature to track 
> document changes over time

Point-in-time queries <https://docs.marklogic.com/guide/app-dev/point_in_time> 
are not designed for versioning, as I think you're describing it. The 
timestamps are internal bookkeeping. (Think of them as monotonically increasing 
integers rather than wall clock readings.) Querying at specific timestamp 
relies on _not_ merging deleted fragments. For short windows, like minutes or 
even hours, depending on your workload, this is OK. However, merging is 
necessary and useful to maintain the health of a database.

Can you elaborate what you mean by "maintain the health of a database"?  If 
we'd decide that we never want to delete any data in a certain MarkLogic 
database so that we can roll back to any point in time, what would be the down 
sides?  How would the database become unhealthy?

We have an existing application that makes use of another database system 
(Datomic) exactly in that way, and we would like to carry it over to MarkLogic. 
 The "Inside MarkLogic" document describes point-in-time queries as "Time 
Travel", but what you write seems to say that using timestamps that way would 
be detrimental to the health of the database, so I'd like to learn more before 
we convert.

Thanks!
Hans

--
LambdaWerk GmbH
Oranienburger Straße 87/89
10178 Berlin
Phone: +49 30 555 7335 0
Fax: +49 30 555 7335 99

HRB 169991 B Amtsgericht Charlottenburg
USt-ID: DE301399951
Geschäftsführer:  Hans Hübner

http://lambdawerk.com/


___
General mailing list
General@developer.marklogic.com<mailto:General@developer.marklogic.com>
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Bulk updates (xqsync vs. mlcp)

[Justin Makeig]

> as we want to be able to use the point-in-time query feature to track 
> document changes over time

Point-in-time queries <https://docs.marklogic.com/guide/app-dev/point_in_time> 
are not designed for versioning, as I think you're describing it. The 
timestamps are internal bookkeeping. (Think of them as monotonically increasing 
integers rather than wall clock readings.) Querying at specific timestamp 
relies on _not_ merging deleted fragments. For short windows, like minutes or 
even hours, depending on your workload, this is OK. However, merging is 
necessary and useful to maintain the health of a database. A good use case for 
point-in-time queries is to get a consistent snapshot of query results across 
multiple requests. For example, run a query and get the first page of results. 
Capture the timestamp at which that query ran. Run queries for each subsequent 
page at that initial timestamp until there are no more pages. This allows you 
to split queries across multiple transactions, such as a multi-threaded export. 
(This is, in fact, what mlcp does when employing the snapshot option on an 
export <https://docs.marklogic.com/guide/mlcp/export#id_43184>.)

>  we'd like to avoid reinserting them as we want to be able to use the 
> point-in-time query feature to track document changes over time

You can do this in mlcp with a transform 
<https://docs.marklogic.com/guide/mlcp/import#id_82518>. The transform runs 
server-side. You'll still need to send the data to the server to check the 
hash. (Put a range index on the hash and this check will be speedy.) However, 
you can avoid the indexing/write cost by returning null/empty sequence from 
your ingest transform. You could capture the new hash as part of the same 
transform for documents that have changed. In pseudo-JavaScript:

if(hashExists(doc)) {
  return null;
} else {
  return updateWithHash(doc, calculateHash(doc));
}

Justin

--
Justin Makeig
Director, Product Management
MarkLogic
justin.mak...@marklogic.com

> On Jun 27, 2016, at 9:45 PM, Hans Hübner <hans.hueb...@lambdawerk.com> wrote:
> 
> Hi,
> 
> we're planning to use MarkLogic to do regular bulk updates on a larger set of 
> documents (~1 million).  Many of the documents will be unchanged from their 
> previous version, and we'd like to avoid reinserting them as we want to be 
> able to use the point-in-time query feature to track document changes over 
> time.  I've read an old thread in this forum that suggested calculating a 
> checksum over each input document and then only writing it to the database if 
> the previous version's checksum differs.  In that same thread, it was also 
> suggested that xqsync could be used.
> 
> Now xqsync apparently was replaced by mlcp, and I can find an indication in 
> the mlcp documentation that it avoids writing unchanged documents.
> 
> Can anyone suggest the best way to approach this?
> 
> Thanks!
> Hans
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] POST multiple documents in a single request

[Justin Makeig]

Use xdmp:multipart-decode(), passing in your boundary ('93e70dee8807c5e7'). 
(You usually specify the boundary identifier in the request Accept header.) 
xdmp:multipart-decode() will give you a sequence of document nodes, prepended 
with a manifest of what it has decoded. It will use the content-type header for 
each part to parse the particular part.

Justin

> On May 17, 2016, at 1:50 PM, Will Thompson  wrote:
> 
> Hi Geert,
> 
> I think I finally have a test that correctly sends multipart/mixed (evidently 
> curl reserves the < character when passing a string value!), but the request 
> body from the POST I get is binary: document{binary{"2d2d2d2d2d2d2 When I 
> send that to the error log, though, it somehow knows to return a string (?):
> 
> --93e70dee8807c5e7
> Content-Disposition: form-data; name="name"; filename="test1.xml"
> Content-Type: application/xml
> 
> 
> Document 1
> --93e70dee8807c5e7
> Content-Disposition: form-data; name="name"; filename="test2.xml"
> Content-Type: application/xml
> 
> 
> Document 2
> ...
> 
> This looks nearly correct, but I'm not sure what to do with this multipart 
> payload now that I have it.
> 
> -W
> 
> 
>> On May 17, 2016, at 1:19 PM, Geert Josten  wrote:
>> 
>> Hi Will,
>> 
>> Have you tried using an HTTP client that supports sending multipart/mixed?
>> You should also be able to use form-data. That works in a similar way, and
>> you can test that with a HTML form with multiple file input fields. These
>> input fields should get translated to request-fields with file names
>> attached.
>> 
>> Cheers,
>> Geert
>> 
>> On 5/17/16, 8:09 PM, "general-boun...@developer.marklogic.com on behalf of
>> Will Thompson" > wthomp...@oconnors.com> wrote:
>> 
>>> Is there a recommended method of posting multiple documents to a ML
>>> endpoint in one request? Maybe this is a more general HTTP question, but
>>> I have tried several permutations of curl options, and unfortunately the
>>> only one that works on the ML side involves joining the documents
>>> together into a single payload with multiple root elements:
>>> 
>>> curl -u user:pass -k -X POST \
>>> -H "Content-Type: text/xml" \
>>> -d 'Document 1Document 2' \
>>> https://localhost:1234/my-endpoint
>>> 
>>> And in XQuery, the sequence of  elements is retrieved by
>>> xdmp:get-request-body()/node(). But this can't be right, can it?
>>> 
>>> -Will
>>> ___
>>> General mailing list
>>> General@developer.marklogic.com
>>> Manage your subscription at:
>>> http://developer.marklogic.com/mailman/listinfo/general
>> 
>> ___
>> General mailing list
>> General@developer.marklogic.com
>> Manage your subscription at: 
>> http://developer.marklogic.com/mailman/listinfo/general
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] POST multiple documents in a single request

[Justin Makeig]

The REST Client API uses HTTP multipart to do batch inserts 
<http://docs.marklogic.com/guide/rest-dev/bulk>. You can do the same from your 
own HTTP app server too. Take a look at 
<http://docs.marklogic.com/xdmp:multipart-decode>.

Justin

--
Justin Makeig
Director, Product Management
MarkLogic


> On May 17, 2016, at 11:09 AM, Will Thompson <wthomp...@oconnors.com> wrote:
> 
> Is there a recommended method of posting multiple documents to a ML endpoint 
> in one request? Maybe this is a more general HTTP question, but I have tried 
> several permutations of curl options, and unfortunately the only one that 
> works on the ML side involves joining the documents together into a single 
> payload with multiple root elements:
> 
> curl -u user:pass -k -X POST \
>  -H "Content-Type: text/xml" \
>  -d 'Document 1Document 2' \
>  https://localhost:1234/my-endpoint
> 
> And in XQuery, the sequence of  elements is retrieved by 
> xdmp:get-request-body()/node(). But this can't be right, can it?
> 
> -Will
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general







smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] JS: require() path resolution

[Justin Makeig]

Florent,
What is your modules root in your app server configuration? Do you have a 
'/lib' at the end?

The import rules are the same in both XQuery and JavaScript 
.

1. When an import/invoke/spawn path starts with a leading slash, first look 
under the Modules directory
2. If the import/invoke/spawn path starts with a slash, and it is not found 
under the Modules directory, then start at the App Server root.
3. If the import/invoke/spawn path does not start with a slash, first look 
under the Modules directory. If the module is not found there, then look 
relative to the location of the module that called the function.

Please work through support  to file a bug if the 
behavior you’re seeing doesn’t adhere to the above rules.

Justin


> On Apr 28, 2016, at 3:20 AM, Florent Georges  wrote:
> 
> Hi Geert,
> 
> Thank you for this.  Got any response?
> 
> -- 
> Florent Georges
> http://fgeorges.org/
> http://h2oconsulting.be/
> 
> 
> On 5 April 2016 at 11:27, Geert Josten wrote:
>> I forwarded your message internally..
>> 
>> Cheers,
>> Geert
>> 
>> On 4/5/16, 11:08 AM, "general-boun...@developer.marklogic.com on behalf of
>> Florent Georges" > li...@fgeorges.org> wrote:
>> 
>>> Just tried 8-0.5, still the same behaviour.  I don't have access to
>>> Discuss, I will try to find whether there is a better way to report it
>>> if Developer Discussion is not well suited.
>>> 
>>> Regards,
>>> 
>>> --
>>> Florent Georges
>>> http://fgeorges.org/
>>> http://h2oconsulting.be/
>>> 
>>> 
>>> On 5 April 2016 at 10:50, Geert Josten wrote:
 Try with 8.0-5, and if it fails on that version as well, mail discuss or
 file a bug..
 
 Cheers
 
 On 4/5/16, 10:45 AM, "general-boun...@developer.marklogic.com on behalf
 of
 Florent Georges"  wrote:
 
> Hi Geert,
> 
> No, it's imported from the dir within which lib/ is.
> 
> Given the different error messages, it seems there is something wrong
> with the resolution.  The following examples (if passed to require()
> in the code below) resolve to (from the error message):
> 
> - "lib/docs.xqy" -> "lib/lib/docs.xqy"
> - "./lib/docs.xqy" -> "./lib/lib/docs.xqy"
> - "docs.xqy" -> "docs.xqy"
> - "./docs.xqy" -> "./docs.xqy"
> 
> Regards,
> 
> --
> Florent Georges
> http://fgeorges.org/
> http://h2oconsulting.be/
> 
> 
> On 5 April 2016 at 10:31, Geert Josten wrote:
>> Are you invoking the require from code running at lib/ itself? Maybe
>> you
>> meant to require /lib/docs.xqy instead?
>> 
>> Cheers,
>> Geert
>> 
>> On 4/5/16, 10:28 AM, "general-boun...@developer.marklogic.com on
>> behalf
>> of
>> Florent Georges" > li...@fgeorges.org> wrote:
>> 
>>> Hi,
>>> 
>>> In JavaScript, I have a module importing (trying to import) an XQuery
>>> module:
>>> 
>>>   var d = require('lib/docs.xqy');
>>> 
>>> But that throws the error: "XDMP-MODNOTFOUND: var d =
>>> require('lib/docs.xqy'); -- Module lib/lib/docs.xqy not found".
>>> 
>>> See the "lib/lib/docs.xqy"? (with double "lib/")
>>> 
>>> If I change it to "require('docs.xqy')", it complains (correctly) that
>>> "Module docs.xqy not found".
>>> 
>>> Did I miss anything?
>>> 
>>> Regards,
>>> 
>>> --
>>> Florent Georges
>>> http://fgeorges.org/
>>> http://h2oconsulting.be/
>>> ___
>>> General mailing list
>>> General@developer.marklogic.com
>>> Manage your subscription at:
>>> http://developer.marklogic.com/mailman/listinfo/general
>> 
>> ___
>> General mailing list
>> General@developer.marklogic.com
>> Manage your subscription at:
>> http://developer.marklogic.com/mailman/listinfo/general
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
 
 ___
 General mailing list
 General@developer.marklogic.com
 Manage your subscription at:
 http://developer.marklogic.com/mailman/listinfo/general
>>> ___
>>> General mailing list
>>> General@developer.marklogic.com
>>> Manage your subscription at:
>>> http://developer.marklogic.com/mailman/listinfo/general
>> 
>> ___
>> General mailing list
>> General@developer.marklogic.com
>> Manage your subscription at:

Re: [MarkLogic Dev General] XDMP-ENTITYREF: (err:XPST0003) Invalid entity reference "Psm#"

[Justin Makeig]

You need to escape your & as  in XML.


> On Mar 7, 2016, at 12:19 PM, Danny Sinang  wrote:
> 
> I'm trying to call xdmp:http-get with the following option :
> 
> declare variable $options :=  
>  
>   
> Bearer 
> TBz#cGDedppn4QdH&93Ra(7w@$!9LsGG5NqiJ^)PTh99GW6!s*O2*tz4oGR8
> 
> ;  
> 
> But I keep getting XDMP-ENTITYREF: (err:XPST0003) Invalid entity reference 
> "Psm#" .
> 
> Do I need to escape the # character ?
> 
> If so, how ?
> 
> Regards,
> Danny
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Dynamically Create View Based On Indexes

[Justin Makeig]

You can import XQuery libraries into JavaScript modules and call functions as 
if they were JavaScript: 

'use strict';
var admin = require('/MarkLogic/admin');
var config = admin.getConfiguration();
admin.databaseGetRangeElementIndexes(config, xdmp.database());

Justin


--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com

> On Feb 2, 2016, at 8:50 AM, Ashley Peacock <apeac...@elevate.co.uk> wrote:
> 
> Thanks for the reply.
>  
> I’d love to use the JS library instead, but I couldn’t find the JS functions 
> that would replicate the ones below – such as 
> admin:database-get-range-element-indexes. Maybe I’m looking in the wrong 
> place?
>  
> Ashley
>  
> From: general-boun...@developer.marklogic.com 
> <mailto:general-boun...@developer.marklogic.com> 
> [mailto:general-boun...@developer.marklogic.com 
> <mailto:general-boun...@developer.marklogic.com>] On Behalf Of Christopher 
> Hamlin
> Sent: 02 February 2016 16:43
> To: MarkLogic Developer Discussion
> Subject: Re: [MarkLogic Dev General] Dynamically Create View Based On Indexes
>  
>  
>  
> On Tue, Feb 2, 2016 at 10:55 AM, Ashley Peacock <apeac...@elevate.co.uk 
> <mailto:apeac...@elevate.co.uk>> wrote:
> Hey,
>  
> I have the basic idea of how to achieve creating a view based on the current 
> indexes on a field, I just can’t tie all the pieces together so hoping 
> someone can help!
>  
> When calling admin:database-get-range-element-indexes() it returns a bunch of 
> XML defining the indexes. I want to loop through each index, retrieve some 
> values (the XML looks something like:
>  
> 2016-02-02 14:14:28.322 Info: TaskServer:  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance 
> <http://www.w3.org/2001/XMLSchema-instance>" 
> xmlns="http://marklogic.com/xdmp/database 
> <http://marklogic.com/xdmp/database>">
> 2016-02-02 14:14:28.322 Info: TaskServer:   int
> 2016-02-02 14:14:28.322 Info: TaskServer:   
> 2016-02-02 14:14:28.322 Info: TaskServer:   
> 2016-02-02 14:14:28.322 Info: TaskServer:   localname
> 2016-02-02 14:14:28.322 Info: TaskServer:   
> true
> 2016-02-02 14:14:28.322 Info: TaskServer:   
> ignore
> 2016-02-02 14:14:28.322 Info: TaskServer: 
>  
> And store then in a sequence. I’ve tried the below to no luck, as my XQuery 
> is not the strongest.
>  
> Let $seq := ()
> For $i in admin:database-get-range-element-indexes(admin:get-configuration(), 
> xdmp:database(‘myDatabase’))
> Return fn:insert-before($seq, 0, 
> view:column($i/range-element-index/localname, 
> $i/range-element-index/localname)
>  
> But that doesn’t seem to work (there’s no error, it just doesn’t “do” 
> anything).
>  
>  
> The insert-before is returning a new sequence each time, so not what you 
> expect (doesn't change $seq). You aren't using the namespace for the index 
> elements either, so wouldn't be finding anything.
>  
> Something like the below is probably closer.  It gets the 
> namespace/localname, tokenizes since you can have more than one, removes a 
> hyphen since that's not allowed in a view column name.  Did sort-of work on 
> the simplistic basic db I tested on, but may be more complications.
>  
> xquery version "1.0-ml";
> import module namespace admin = "http://marklogic.com/xdmp/admin 
> <http://marklogic.com/xdmp/admin>" at "/MarkLogic/admin.xqy";
> import module namespace view = "http://marklogic.com/xdmp/view 
> <http://marklogic.com/xdmp/view>" at "/MarkLogic/views.xqy";
> declare namespace db = 'http://marklogic.com/xdmp/database 
> <http://marklogic.com/xdmp/database>';
> let $config := admin:get-configuration()
> let $seq := 
>   for $i in 
> admin:database-get-range-element-indexes(admin:get-configuration(), 
> xdmp:database('Documents'))
>   let $namespace-uri := $i/db:namespace-uri/fn:string()
>   for $localname in fn:tokenize ($i/db:localname/fn:string(), '[ ,]+')
>   let $col-name := fn:replace ($localname, '-', '')
>   let $col := view:column($col-name, cts:element-reference (fn:QName 
> ($namespace-uri, $localname)))
>   return $col
> return $seq
>  
> It looks like it gives what you want, though I didn't use the results at all.
>  
> Also, if JS is more natural, you can use that instead.
>  
> =ch
>  
>  
> 
> Click here 
> <https://www.mailcontrol.com/sr/YxyAmhc2!bzGX2PQPOmvUhkLFoJbzkFEjyrqhuh9QPViZjnwTtxSFEksa2wfyPQswHpDQtigvFDaNxM9hG2QrA==>
>  to report this email as spam.
> 
> 
>  
> 
> 
> 
> 
> Ashley Peacock
> 
> Developer II
> Elevate Credit International Limited
> P: 01284 717800 | Ext: 877 |  Elevate.co.uk <h

Re: [MarkLogic Dev General] cts:element-range-query

[Justin Makeig]

That query will be problematic if you have multiple Date elements in your 
documents. You should transform your data to store a proper xs:date.

Justin

--
Justin Makeig
Director, Product Management
MarkLogic

> On Jan 21, 2016, at 3:06 AM, rajamani.marimu...@cognizant.com wrote:
> 
> Hi  Pragya,
>  
> I don’t  think ,you can use the below format for date range comparison . 
> Better you have same date details in attribute too. So , simply you can setup 
> element attribute range index with integer option then try the following code 
> . It will work.
>  
>  
> let $fromDate := ("2011-03-04")
> let $toDate := ("2011-06-08")
> let $docs := cts:search(fn:doc()/DocumentList/Document,
>cts:and-query((
> 
> cts:element-attribute-range-query(xs:QName("Date"),xs:QName("Year"), 
> ">=",xs:int(substring($fromDate,1,4))),
>  
> cts:element-attribute-range-query(xs:QName("Date"),xs:QName("Year"), 
> "<=",xs:int(substring($toDate,1,4))),
>  
> cts:element-attribute-range-query(xs:QName("Date"),xs:QName("Month"), 
> ">=",xs:int(substring($fromDate,6,2))),
>  
> cts:element-attribute-range-query(xs:QName("Date"),xs:QName("Month"), 
> "<=",xs:int(substring($toDate,6,2))),
>  
> cts:element-attribute-range-query(xs:QName("Date"),xs:QName("Day"), 
> ">=",xs:int(substring($fromDate,9,2))),
>  
> cts:element-attribute-range-query(xs:QName("Date"),xs:QName("Day"), 
> "<=",xs:int(substring($toDate,9,2)))
> ))
>   )
> return $docs
>  
>  
> By
> Raja>>>
>  
> From: general-boun...@developer.marklogic.com 
> <mailto:general-boun...@developer.marklogic.com> 
> [mailto:general-boun...@developer.marklogic.com 
> <mailto:general-boun...@developer.marklogic.com>] On Behalf Of Kapoor, Pragya
> Sent: Thursday, January 21, 2016 2:52 PM
> To: MarkLogic Developer Discussion
> Subject: [MarkLogic Dev General] cts:element-range-query
>  
>  
> Hi,
>  
> I need to query an element(Date) using cts:element-range-query, but the value 
> this element(Date) is in different format.
>  
> 
> 2011 Mar 04
> 
>  
>  
> query :
>  
>   let $fromDate := xs:date("2011-04-08")
>   let $toDate := xs:date("2011-06-08")
>   let $docs := cts:search(fn:doc("/misc/DocList.xml")/DocumentList/Document,
>cts:and-query((
> cts:element-range-query(xs:QName("Date"), 
> "<=",$fromDate),
>  cts:element-range-query(xs:QName("Date"), 
> ">=",$toDate)
> ))
>   )
>  return $docs
>  
> How can run this query or I need to use some other api?
>  
> Thanks
> Pragya
> 
> 
> "This e-mail and any attachments transmitted with it are for the sole use of 
> the intended recipient(s) and may contain confidential , proprietary or 
> privileged information. If you are not the intended recipient, please contact 
> the sender by reply e-mail and destroy all copies of the original message. 
> Any unauthorized review, use, disclosure, dissemination, forwarding, printing 
> or copying of this e-mail or any action taken in reliance on this e-mail is 
> strictly prohibited and may be unlawful."
> This e-mail and any files transmitted with it are for the sole use of the 
> intended recipient(s) and may contain confidential and privileged 
> information. If you are not the intended recipient(s), please reply to the 
> sender and destroy all copies of the original message. Any unauthorized 
> review, use, disclosure, dissemination, forwarding, printing or copying of 
> this email, and/or any action taken in reliance on the contents of this 
> e-mail is strictly prohibited and may be unlawful. Where permitted by 
> applicable law, this e-mail and other e-mail communications sent to and from 
> Cognizant e-mail addresses may be 
> monitored.___
> General mailing list
> General@developer.marklogic.com <mailto:General@developer.marklogic.com>
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general 
> <http://developer.marklogic.com/mailman/listinfo/general>



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] XDMP-CHILDLINK error

[Justin Makeig]

Taking a step back from your immediate problem, your document structure looks 
like you're trying to model many atomic entities in a single document. 
Generally in MarkLogic, each atomic item should be its own document. This how 
we've optimized our indexes and I/O. The relational database analog would be 
that documents map to rows, not tables. In your case, you'd have 4,000+ 
individual documents, each with  root elements. (Even better would be to 
give a semantically meaningful name to your node elements, for example, 
"product" or "user".)

If you can provide more detail around what you're trying to accomplish we might 
be able to provide more/better guidance.

Justin


--
Justin Makeig
Director, Product Management
MarkLogic


> On Jan 14, 2016, at 10:36 AM, Prasanth N V R <prasanth.nv...@gmail.com> wrote:
> 
> Hi,
> 
> I have 4000+ nodes in a document. Below is the structure of the document and 
> loaded under "/test/testDocument.xml".
> 
> 
> 
> a
> 1234
> A
> 
>  
> b
> 1234
> B
> 
> 
> c
> 1234
> C
> 
> :
> :
> 
> 
> If a node name matches my input, then I will do a xdmp:node-replace of that.
> let $result := cts:search(my search query goes here)
> return xdmp:node-replace($result/category,newCategory
> 
> When I try to do node replace, am getting error like,
> XDMP-CHILDLINK: (err:XPTY0004) Invalid child link node 
> fn:doc("/test/testDocument.xml")/rootnode/node[257]
> 
> Your help is much appreciated!
> 
> Thanks,
> Prasanth
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Facets on properties in ML4.2

[Justin Makeig]

s/organically/original/

Sigh. Auto-correct.


> On Jan 7, 2016, at 9:50 AM, Justin Makeig <justin.mak...@marklogic.com> wrote:
> 
> Are you really running MarkLogic 4.2? This is ancient. You should plan to 
> upgrade as soon as feasible. It will be very difficult for the community or 
> MarkLogic Support to assist you with this version. The most recent version is 
> 8.0-4.
> 
> It is possible to facet on properties. cts:values has an option that allows 
> you include values from properties fragments. (Or cts:element-values in 4.2.) 
> However, in general, I'd advise against properties fragments. Because they 
> are separate from document fragments, properties require extra work to join 
> at query time. If you don't want to change/enrich your actual document 
> content you can use an envelope to "wrap" your documents:
> 
> 
>   …
>   …
> 
> 
> This has the advantage of segregating the metadata (properties) and data, but 
> also colocates them in the same fragment, reducing the work that queries will 
> have to do.
> 
> Justin
> 
> 
>> On Jan 7, 2016, at 1:35 AM, rajamani.marimu...@cognizant.com 
>> <mailto:rajamani.marimu...@cognizant.com> wrote:
>> 
>> Hi Team,
>>  
>>   We are trying to implement facets from various property elements in ML4.2 
>> version . Is that possible. Like
>>  
>>   Search term :  “sample” present in content body
>>   Facets : on some properties like : author, geography   etc. from document 
>> properties . Kindly appreciate if have some sample custom constraint or 
>> similar options
>>  
>> By
>> Raja >>> 
>>   
>> This e-mail and any files transmitted with it are for the sole use of the 
>> intended recipient(s) and may contain confidential and privileged 
>> information. If you are not the intended recipient(s), please reply to the 
>> sender and destroy all copies of the original message. Any unauthorized 
>> review, use, disclosure, dissemination, forwarding, printing or copying of 
>> this email, and/or any action taken in reliance on the contents of this 
>> e-mail is strictly prohibited and may be unlawful. Where permitted by 
>> applicable law, this e-mail and other e-mail communications sent to and from 
>> Cognizant e-mail addresses may be monitored.
>> ___
>> General mailing list
>> General@developer.marklogic.com <mailto:General@developer.marklogic.com>
>> Manage your subscription at: 
>> http://developer.marklogic.com/mailman/listinfo/general
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Optimizing "order by" with path/field range index

[Justin Makeig]

You can be explicit with cts:index-order() 
.

Justin


> On Oct 30, 2015, at 12:29 PM, Bob H.  wrote:
> 
> All,
> 
> We know that ML will optimize an "order by" clause if it can with a range 
> index. I've done this with element, attribute, and path range indexes on many 
> occasions.  However, I'm faced with a scenario for which I can't seem to find 
> a straightforward optimization.  Your help would be appreciated!
> 
> Consider a database full of documents.  The documents all contain an element 
> with local name "created", but the namespaces vary.  While normalization may 
> be an option for the future, I'm trying to see if there is a way to avoid 
> that for now.
> 
> So, consider the following three documents:
>  bob 2015-10-28T00:00:00Z
>  bob 2015-10-29T00:00:00Z
>  bob 2015-10-30T00:00:00Z
> 
> I can define a path range index using the wildcarded path:
> //*:created
> 
> The path range index clearly works as expected, because I can retrieve all 
> three values above via this cts:values statement:
> cts:values(cts:path-reference('//*:created'))
> 
> However, if I run a cts:search to find the three docs above and order the 
> results by this path, query-trace shows that the sort is not optimized:
> for $item in
> cts:search(doc(), 'bob')
> order by $item//*:created
> return $item
> 
> The query-trace output does not show that a range index was used to optimize 
> the order by:
> 2015-10-30 15:26:01.909 Info: App-Services: at 4:11: Analyzing path for 
> search: fn:doc()
> 2015-10-30 15:26:01.909 Info: App-Services: at 4:11: Step 1 is searchable: 
> fn:doc()
> 2015-10-30 15:26:01.909 Info: App-Services: at 4:11: Path is fully searchable.
> 2015-10-30 15:26:01.909 Info: App-Services: at 4:11: Gathering constraints.
> 2015-10-30 15:26:01.910 Info: App-Services: at 4:11: Search query contributed 
> 1 constraint: cts:word-query("bobh", ("lang=en"), 1)
> 2015-10-30 15:26:01.910 Info: App-Services: at 4:11: Executing search.
> 2015-10-30 15:26:01.910 Info: App-Services: at 4:11: Selected 3 fragments to 
> filter.
> 
> I've tried a number of other approaches to the path range index, including 
> paths that did not contain wildcards but instead included the specific 
> namespace possibilities:
> //(a:created|b:created|c:created)
> /(a:root|b:root|c:root)/(a:created|b:created|c:created)
> 
> I've considered creating a field encompassing the three elements and a field 
> range index, but I don't know of a way to construct the order by clause such 
> that the field range index would be used.  Is that even possible?
> 
> Any suggestions are welcome!  If this sort of approach is a dead end, we will 
> need to wait until we have the opportunity to normalize the data to optimize 
> this.
> 
> Thank you!
> 
> -Bob
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] xdmp:forest-clear

[Justin Makeig]

Yes, my wetware parser missed that. I've filed a bug on your behalf to cover 
this in the docs. Thanks for your help.

Justin

On Oct 5, 2015, at 12:37 AM, Andreas Hubmer 
> wrote:

Thanks for your replies. It would be great if the MarkLogic documentation would 
indicate which operations are not transactional.

@Justin: My code is a multi-statement transaction, hence the updates are 
visible after the ";"



2015-10-02 19:00 GMT+02:00 Wayne Feick 
>:
When you clear a forest in a database you're running against, the clear 
operation is run asynchronously. Your query of test.xml is running prior to the 
clear actually happening.

Wayne.



On 10/02/2015 08:35 AM, Andreas Hubmer wrote:
Hello,

I've found an issue with xdmp:forest-clear.

xdmp:document-insert("test.xml", );
xdmp:forest-clear(xdmp:database-forests(xdmp:database()));
doc("test.xml") (: should be empty :)

When executing the above multi statement transaction, I would expect an empty 
result. But instead  is returned.
When I execute doc("test.xml") manually some moments later, the expected empty 
result is returned.

As a workaround I could use xdmp:document-delete(cts:uris()) but my assumption 
is that xdmp:forest-clear is much faster. Is that true?

Is xdmp:forest-clear somehow asynchronous?
Or is it maybe a visibility problem with deleted data? I run the code as admin 
user (just for testing).

Regards,
Andreas

--
Andreas Hubmer
IT Consultant

EBCONT enterprise technologies GmbH
Millennium Tower
Handelskai 94-96
A-1200 Vienna

Web: http://www.ebcont.com

OUR TEAM IS YOUR SUCCESS

UID-Nr. ATU68135644
HG St.Pölten - FN 399978 d



___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general



--
Wayne Feick
Principal Engineer
MarkLogic Corporation
wayne.fe...@marklogic.com
Phone: +1 650 655 2378
www.marklogic.com

This e-mail and any accompanying attachments are confidential. The information
is intended solely for the use of the individual to whom it is addressed. Any
review, disclosure, copying, distribution, or use of this e-mail communication
by others is strictly prohibited. If you are not the intended recipient, please
notify us immediately by returning this message to the sender and delete all
copies. Thank you for your cooperation.

___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general




--
Andreas Hubmer
IT Consultant

EBCONT enterprise technologies GmbH
Millennium Tower
Handelskai 94-96
A-1200 Vienna

Web: http://www.ebcont.com

OUR TEAM IS YOUR SUCCESS

UID-Nr. ATU68135644
HG St.Pölten - FN 399978 d
___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] xdmp:forest-clear

[Justin Makeig]

A single statement runs at one timestamp. Thus your code sees the database at 
time t0. MarkLogic doesn't actually execute the database updates as they appear 
in your code. Instead, it gathers them up and executes them atomically at the 
end, advancing the timestamp if everything works. Take a look at 
 and especially 
.

Justin


On Oct 2, 2015, at 8:35 AM, Andreas Hubmer 
> wrote:

Hello,

I've found an issue with xdmp:forest-clear.

xdmp:document-insert("test.xml", );
xdmp:forest-clear(xdmp:database-forests(xdmp:database()));
doc("test.xml") (: should be empty :)

When executing the above multi statement transaction, I would expect an empty 
result. But instead  is returned.
When I execute doc("test.xml") manually some moments later, the expected empty 
result is returned.

As a workaround I could use xdmp:document-delete(cts:uris()) but my assumption 
is that xdmp:forest-clear is much faster. Is that true?

Is xdmp:forest-clear somehow asynchronous?
Or is it maybe a visibility problem with deleted data? I run the code as admin 
user (just for testing).

Regards,
Andreas

--
Andreas Hubmer
IT Consultant

EBCONT enterprise technologies GmbH
Millennium Tower
Handelskai 94-96
A-1200 Vienna

Web: http://www.ebcont.com

OUR TEAM IS YOUR SUCCESS

UID-Nr. ATU68135644
HG St.Pölten - FN 399978 d
___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] moving HDFS data to ML

[Justin Makeig]

Where is your bottleneck? You're going to need to provide more information 
about the specific problem you're seeing and what your infrastructure looks 
like. How are the 40M files packaged? HDFS generally does better with large 
aggregate files (like tables), while MarkLogic does better with small, 
individual records (like rows). I can't tell what you're doing or where it's 
going wrong from the sparse detail you've provided below.

> please can some one  let me know how do I  use the  direct access?

Direct access  
is primarily for reading data out of offline MarkLogic archives. Direct access 
is unrelated to mcp's distributed mode. The latter allows you to parallelize 
ingestion over hosts, not just threads. On the other hand, using Direct Access, 
mlcp can read forests off of disk without having to go through a running 
MarkLogic instance. Direct Access does nothing for loading data today. The only 
way to create a MarkLogic forest is through a MarkLogic instance.

Justin


> On Sep 20, 2015, at 7:52 PM, manju rajendran  wrote:
> 
> Hi
> 
> I have a  load task   of  40 million files fromHDFS(Hadoop)  to the  ML DB
> 
> 
> Architecturally  the  ML server is  configured as a  file I/O server,  MLCP  
> is loading the   data in  Distributed mode, the  Threads  used in the  MLCP 
> (Shell scripts)  did ** not ** improve performance.
> 
> please can some one  let me know how do I  use the  direct access?.
> 
> Thanks
> Manju Rajendran
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Query on Fragmentation

[Justin Makeig]

The general rule of fragmentation is "Don't use fragmentation". It is a very 
sharp tool for a very specific use case, where you have large logical sections, 
like chapters, under a parent document. What we've found generally is that 
taking the time to model and split aggregated "fragments" into explicit 
documents is a better approach 9.9 times out of 10.

Can you say a little more about what you're trying to accomplish? What do your 
queries look like?

Justin

> On Sep 16, 2015, at 3:30 AM, Pavadaidurai A  
> wrote:
> 
> Dear All,
>  
> I have taken the below example from Marklogic documentation. I have couple of 
> questions reg Fragmentation. I understand from below example that “Citation”, 
> will need to defined as the fragement root
>  
> 1)  Assuming that we few other elements under CitationSet besides 
> Citation element, what happens to the Misc elemens? In other words, if I am 
> using Fragment root, what happens to the left over elements in the document?
> 2)  I understand that after setting Fragmentation rule, existing 
> documents will remain unfragmented unless re-indexed (or re-loaded). Does 
> xdmp:document-set-collection API, takes care of fragmenting the document. I 
> am asking this question because xdmp:document-set-collection API internally 
> rewrites the document into the database.
> 3)  Is there anyway to find out if a given document is fragmented or not?
>  
> 
> citation1
> citation2
> citation3
> citation4
> citation5
> misc1
> misc2
> misc3
> 
>  
>  
> Fragment Roots 
> If a document contains many instances of an XML structure that share a common 
> element name, then these structures make sensible fragments. With MarkLogic 
> Server, you can use this common element name as a fragment root.
> 
> The following diagram shows an XML document rooted at  that 
> contains many instances of a  node. Each  node contains 
> further XML and averages between 15K and 20K in size. Based on this 
> information,  is a sensible element to use as a fragment root:
> 
> 
> 
>  
>  
> Thanks,
> Durai.
>  
>  CAUTION - Disclaimer *
> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely 
> for the use of the addressee(s). If you are not the intended recipient, 
> please 
> notify the sender by e-mail and delete the original message. Further, you are 
> not 
> to copy, disclose, or distribute this e-mail or its contents to any other 
> person and 
> any such actions are unlawful. This e-mail may contain viruses. Infosys has 
> taken 
> every reasonable precaution to minimize this risk, but is not liable for any 
> damage 
> you may sustain as a result of any virus in this e-mail. You should carry out 
> your 
> own virus checks before opening the e-mail or attachment. Infosys reserves 
> the 
> right to monitor and review the content of all messages sent to or from this 
> e-mail 
> address. Messages sent to or from this e-mail address may be stored on the 
> Infosys e-mail system.
> ***INFOSYS End of Disclaimer INFOSYS***
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] global variable usage

[Justin Makeig]

Or use the database. You have transactional global state just below your code. 
If you need to isolate application artifacts from data in a database, you can 
use document permissions and amp the function that reads/modifies the state, 
such that normal application code wouldn't ever see it.

Justin


> On Sep 1, 2015, at 2:19 AM, Geert Josten  wrote:
> 
> Keep in mind though, server and session fields are tied to one host only. If 
> you want to share values across a database, easiest is to just put it in the 
> database..
> 
> Cheers
> 
> From:  > on behalf of Florent 
> Georges >
> Reply-To: MarkLogic Developer Discussion  >
> Date: Tuesday, September 1, 2015 at 10:43 AM
> To: MarkLogic Developer Discussion  >
> Subject: Re: [MarkLogic Dev General] global variable usage
> 
>   Hi,
> 
>   I am not sure about the details of what you need to achieve, but it sounds 
> like you want either Server or Session Fields:
> 
> https://docs.marklogic.com/xdmp:set-server-field 
> 
> https://docs.marklogic.com/xdmp:set-session-field 
> 
> 
>   Regards,
> 
> -- 
> Florent Georges
> http://fgeorges.org/ 
> http://h2oconsulting.be/ 
> 
> 
> On 1 September 2015 at 09:52, sweet frd wrote:
> Hi All,
> 
>   I need to set/get/maintain a variable value across different xquery 
> call. I am declaring a variable in one module xquery file and importing the 
> same in other xquery and set the variable value.
> 
> In case of multiple calls to the xquery, will the value of the variable ll 
> get assigned depend on the particular call or like java static the value will 
> get changed based on the next call.
> 
> Is the xquery variable is static or normal variable depends on the xquery 
> call.??
> 
> Thanks,
> N. Sumathi.
> 
> ___
> General mailing list
> General@developer.marklogic.com 
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
> 
> -- 
>  Florent Georges
>  
> http://fgeorges.org/ 
> 
> http://h2oconsulting.be/ 
> 
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Search in first paragraph or portion of the documents

[Justin Makeig]

With 50GB XML documents you're going to have many other problems. To couch this 
in terms of a relational database, that's like having a 50GB row in a table. In 
both cases, there's probably a better way to model your data to fit with the 
I/O and indexing patterns of the database. 

Are these 50GB documents actually many different documents aggregated into one 
big container? If so, you'll be much better off splitting them into individual 
documents in the 10K–100K range, plus or minus one order of magnitude—the 
equivalent of a Debit Entry versus a General Ledger; an Article versus a 
Magazine; an Animal versus a Zoo; etc. You're going to have to tell us a little 
more about your data and queries in order to recommend something more specific, 
though.

Justin


 On Aug 27, 2015, at 12:12 PM, Yang, Yun yun.y...@wolterskluwer.com wrote:
 
 Thanks Justin for the suggestions. So for the smaller docs, the solution will 
 work. What happen the docs we have are big docs, for example, over 50 GB, so 
 create a new element inside the same big document would have an issue for 
 opening and snippet, may be create a separate doc to hold the first portion 
 of the doc?
  
 Any suggestions of how to handle big doc?
  
 Thanks,
  
 Yun
  
 From: general-boun...@developer.marklogic.com 
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of Justin Makeig
 Sent: Thursday, August 27, 2015 2:02 PM
 To: MarkLogic Developer Discussion
 Subject: Re: [MarkLogic Dev General] Search in first paragraph or portion of 
 the documents
  
 Wrap the first portion in a new element (assuming you're talking about XML 
 here). The you can use something like cts:element-query 
 http://docs.marklogic.com/cts:element-query 
 http://docs.marklogic.com/cts:element-query or cts:element-word-query 
 http://docs.marklogic.com/cts:element-word-query 
 http://docs.marklogic.com/cts:element-word-query to restrict queries to 
 just that element. Think of the XML elements as a way to tell MarkLogic which 
 specific parts of the document to index.
  
 Justin
  
 On Aug 27, 2015, at 11:57 AM, Yang, Yun yun.y...@wolterskluwer.com 
 mailto:yun.y...@wolterskluwer.com wrote:
  
 All,
  
 We have 20 million documents, there is an use case where we must search only 
 the first portion of each document. Is there a way to do that? The first 
 portion of a document is defined as first 50 words or 100 words, etc.
  
 Thanks,
  
 Yun
  
 ___
 General mailing list
 General@developer.marklogic.com mailto:General@developer.marklogic.com
 Manage your subscription at: 
 http://developer.marklogic.com/mailman/listinfo/general 
 http://developer.marklogic.com/mailman/listinfo/general
  
 ___
 General mailing list
 General@developer.marklogic.com
 Manage your subscription at: 
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Search in first paragraph or portion of the documents

[Justin Makeig]

Wrap the first portion in a new element (assuming you're talking about XML 
here). The you can use something like cts:element-query 
http://docs.marklogic.com/cts:element-query or cts:element-word-query 
http://docs.marklogic.com/cts:element-word-query to restrict queries to just 
that element. Think of the XML elements as a way to tell MarkLogic which 
specific parts of the document to index.

Justin

 On Aug 27, 2015, at 11:57 AM, Yang, Yun yun.y...@wolterskluwer.com wrote:
 
 All,
  
 We have 20 million documents, there is an use case where we must search only 
 the first portion of each document. Is there a way to do that? The first 
 portion of a document is defined as first 50 words or 100 words, etc.
  
 Thanks,
  
 Yun
  
 ___
 General mailing list
 General@developer.marklogic.com
 Manage your subscription at: 
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Failure to enforce occurrence indicators in function signature

[Justin Makeig]

You're likely running into function mapping 
http://docs.marklogic.com/guide/xquery/enhanced#id_55459.

Justin


 On Aug 21, 2015, at 9:44 AM, Hasegawa, Margarete 
 margarete.haseg...@ptsem.edu wrote:
 
 Hi, 
 
 Has anyone experienced an issue with MarkLogic failing to enforce occurrence 
 indicators in function calls? For example, a function expects xs:string? 
 (meaning that the function is expecting either the empty sequence or a single 
 node), but instead accepts multiple values and does not fail. We've observed 
 that instead of raising a syntax error, ML will sometimes give unpredictable 
 behavior instead. Could this be a bug?
 
 Many thanks!
 Maggie 
 
 ___
 General mailing list
 General@developer.marklogic.com
 Manage your subscription at: 
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] path range index

[Justin Makeig]

Take a look at the example in the cts:values documentation 
http://docs.marklogic.com/cts:values. (Spoiler: You'll need a 
cts:path-reference  http://docs.marklogic.com/cts:path-reference.)

Justin


 On Aug 18, 2015, at 10:27 AM, Paul M pjm...@yahoo.com wrote:
 
 Suppose I want all the values for a specific path range index.
 Normally, I would use cts:element-values and cts:frequencey on a range index. 
 How would I do this on a path range index, if possible.
 
 
 Thank You.
 ___
 General mailing list
 General@developer.marklogic.com
 Manage your subscription at: 
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Not ask for credentials on subsequent HTTP requests

[Justin Makeig]

That's the job of the client with HTTP auth. Which client are you using? What 
does your XHR request look like? Is there CORS involved? HTTPS? A password 
manager?

Justin

 On Jul 30, 2015, at 10:00 AM, Danny Sinang d.sin...@gmail.com wrote:
 
 I have an ML app on port 7600 which prompts me for credentials, which is fine.
 
 But the same app makes subsequent HTTP service calls to fetch data via Ajax 
 from the same port. Each of these subsequent calls will prompt for a username 
 and password unless I hard-code the credentials in my XQuery script.
 
 Is there a way to tell ML to just reuse the credentials I initially provided 
 so I won't have to reenter them or hard code them in ?
 
 Regards,
 Danny
 
 ___
 General mailing list
 General@developer.marklogic.com
 Manage your subscription at: 
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] mlcp ability to skip corrupt zip files?

[Justin Makeig]

Can you please contact MarkLogic Support 
supp...@marklogic.commailto:supp...@marklogic.com on this? There might be a 
bug here.

I'll be sure to update this list with what we find.

Thanks.

Justin

On Jul 21, 2015, at 9:58 AM, Morales-Martin, Kristina 
kmorales-mar...@cas.orgmailto:kmorales-mar...@cas.org wrote:


Dear all,

We are using the MarkLogic Content Pump to push content from many directories 
that have zip files that in turn contain .xml files.
From the last communication with Geet, we are also using the transform option 
in order to ingest only xml content.  This suggested filtering approach
using a transform works.

Unfortunately, when mlcp encounters a corrupt zip file (which we possibly can 
get from our sources),
the process terminates.  Is there an option to instruct mlcp to keep going, 
that is, to skip the corrupt skip file, and continue processing the large and
deeply nested directories for the rest of the zip files?  It looks like the 
-tolerate_errors option won’t work given that we need to use a transform to 
ingest only xml files,
and that forces the batch size to 1.

Please advise.

We are using the following options:
-input_file_path $inputFilePath \
-mode local -input_compressed true \
-output_uri_replace 
(\/.+\/+)(?=.+\.zip),'/ourOverrideOfTheURIToRemoveTheLeadingNASPath/' \
-output_collections $collections \
-database $dbName -output_permissions …
-transform_module /ourNamespace/ourTransformModule.xqy  \
-transform_namespace http://cas.org/...; \
-xml_repair_level full \

Thank you,

Kristina Morales-Martin
Sr. Technical Information Specialist, e-Content Operations
CAS, a division of the American Chemical Society
2540 Olentangy River Road
Columbus, OH 43202
Phone: 614-447-3600, ext. 2322
Fax: 614-447-3827
www.cas.orghttp://www.cas.org/


Confidentiality Notice: This electronic message transmission, including any 
attachment(s), may contain confidential, proprietary, or privileged information 
from Chemical Abstracts Service (“CAS”), a division of the American Chemical 
Society (“ACS”). If you have received this transmission in error, be advised 
that any disclosure, copying, distribution, or use of the contents of this 
information is strictly prohibited. Please destroy all copies of the message 
and contact the sender immediately by either replying to this message or 
calling 614-447-3600.

___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] JSON with namespace (XML to JSON - JSON to XML)

[Justin Makeig]

No, for better or worse, JSON doesn't have the concept of namespaces as part of 
the data model like XML. You could capture the XML namespace as a property of 
the JSON document during transformation or use a naming convention for property 
names, e.g. person$firstName. For namespaces that cover the entire document 
you might also consider a MarkLogic collection, for example,

person xmlns=http://example.org/ns/person 
http://example.org/ns/personfirst-nameGnanaprakash/first-name/person 

would map to 

{ firstName: Gnanaprakash } in the collection 
http://example.com/ns/person#Person http://example.com/ns/person#Person, 
capturing the fully namespaced root element.

Justin



 On May 5, 2015, at 6:12 AM, gnanaprakash.bodire...@cognizant.com wrote:
 
 Hi
  
 I am just wondering is there any way we have JSON documents with namespaces.
  
 When an XML document is converted as JSON we lose the namespaces as JSON 
 won’t carry namespace
  
 and we try to covert JSON back to XML, the XML no longer carry the namespace 
 as the source is JSON.
  
 Is there any solution for this?
  
 Regards,
 Gnanaprakash Bodireddy
 
  
  
  
 This e-mail and any files transmitted with it are for the sole use of the 
 intended recipient(s) and may contain confidential and privileged 
 information. If you are not the intended recipient(s), please reply to the 
 sender and destroy all copies of the original message. Any unauthorized 
 review, use, disclosure, dissemination, forwarding, printing or copying of 
 this email, and/or any action taken in reliance on the contents of this 
 e-mail is strictly prohibited and may be unlawful. Where permitted by 
 applicable law, this e-mail and other e-mail communications sent to and from 
 Cognizant e-mail addresses may be monitored.
 ___
 General mailing list
 General@developer.marklogic.com
 Manage your subscription at: 
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] question about xdmp:encoding-language-detect

[Justin Makeig]

Jakob,
Are there any other markers that are specific to your domain that could help 
you triangulate? The built-in detection doesn't (and can't) know the context of 
your business. Some pre- or post-detection analysis might help you to better 
narrow. For example, is a specific source known to not have Croatian or Serbian 
content, but might have Latvian? Are there entities (e.g. names, addresses, 
etc.) that are decent indicators of Latvian? I don't know the specifics of your 
app or content, but there might be other context that you could pull in to 
enhance the out-of-the-box identification.

Justin


--
Justin Makeig
Director, Product Management
MarkLogic
justin.mak...@marklogic.com
+1 (650) 655-2387

 On Mar 27, 2015, at 8:44 AM, Jakob Fix jakob@gmail.com wrote:
 
 Thanks Mary for your quick reply. It's an explanation that I
 understand, but this doesn't resolve my initial problem.
 Any idea how to solve this in the short term and whether there are
 improvements in the pipeline? Or that it's not a high priority?
 
 cheers,
 Jakob.
 
 
 On Fri, Mar 27, 2015 at 4:34 PM, Mary Holstege
 mary.holst...@marklogic.com wrote:
 On Fri, 27 Mar 2015 08:23:19 -0700, Jakob Fix jakob@gmail.com wrote:
 
 Hello, I think this message got lost when the mailing list was down in
 February (or nobody has an answer ...)
 
 Thanks,
 Jakob.
 
 The xdmp:encoding-language-detect uses the ICU libraries to do the
 detection. Serbian and Croatian are very closely related to each other and
 have some similar orthography to Latvian (although not a great deal of
 linguistic similarity, it must be said). I think the ICU libraries
 probably lack some of the linguistic sophistication of Google's backend.
 
 It has nothing to do with the licensing options.
 
 //Mary
 
 
 -- Forwarded message --
 From: Jakob Fix jakob@gmail.com
 Date: Sat, Feb 28, 2015 at 10:59 PM
 Subject: question about xdmp:encoding-language-detect
 To: General Mark Logic Developer Discussion
 general@developer.marklogic.com
 
 
 Hello,
 
 using ML7.0-3, the above function, given more than 3500 characters of
 Latvian news story text, returns Croatian twice and Serbian once in
 the top three results:
 
 encoding-language xmlns=xdmp:encoding-language-detect
  encodingutf-8/encoding
  languagehr/language
  score7.081/score
 /encoding-language
 encoding-language xmlns=xdmp:encoding-language-detect
  encodingutf-8/encoding
  languagehr/language
  score7.012/score
 /encoding-language
 encoding-language xmlns=xdmp:encoding-language-detect
  encodingutf-8/encoding
  languagesr/language
  score6.882/score
 /encoding-language
 ...
 
 and no Latvian in sight. Google translate as well as
 detectlanguage.com correctly and with sufficient self-assurance return
 the correct result.
 
 Can someone explain what the reason behind this lack of confidence and
 the wrong detection is? Do you need the right language pack (I'm
 playing around with the developer licence which I thought is
 full-featured)? Is this something that needs training? The doc doesn't
 say so.
 
 Thanks!
 
 cheers,
 Jakob.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 
 --
 Using Opera's revolutionary email client: http://www.opera.com/mail/
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general






smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] ML8 Json

[Justin Makeig]

Paul,
You're constructing XML and trying to insert an object-node child. That's not 
allowed in MarkLogic 8. Previously xdmp:to-json returned a string, as there was 
no object-node type. In MarkLogic 8 we introduced new node types to represent 
JSON: object-node, array-node, null-node, boolean-node, and number-node. (JSON 
strings share text node types with their XML brethren.) In order to get the 
serialized string you should use xdmp:quote, as you suggest.

Justin 


--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com

 On Mar 23, 2015, at 2:45 PM, Paul Williams 
 paul.willi...@flatironssolutions.com wrote:
 
 In upgrading from ML6 to ML8 we have an issue in our application code where 
 we are generating JSON for a client-side script.  I know there has been a 
 huge change in JSON support, but can anyone tell me exactly what is going on 
 here?
  
 Sample code:
 xquery version 1.0-ml;
 let $obj :=
json:object xmlns:json=http://marklogic.com/xdmp/json;
json:entryjson:keyname/json:keyjson:value 
 xsi:type=xs:stringworld/json:value/json:entry
json:entryjson:keylabel/json:keyjson:value 
 xsi:type=xs:stringhello/json:value/json:entry
/json:object
 let $doc := xdmp:to-json(json:object($obj))
 return
script type=text/javascriptvar obj = {$doc}/script
  
 If you execute that in ML6, as expected, you get…
 script type=text/javascriptvar obj = {name:world, 
 label:hello}/script
  
 But in ML8, you get this error pointing at the highlighted reference
 XDMP-CHILDNODEKIND: $doc -- element nodes cannot have object 
 node children
  
 I’m not sure exactly what is the “element node” this message refers to.  I 
 know that I can quote the reference (xdmp:quote($doc)) and make it work, but 
 I’m trying to understand what specifically is different between ML6  ML8 
 that is causing this problem so I can be sure of my workaround. 
 Any comments?
 TIA
  
 Paul Williams | Technical Consultant
 M: +1 402 203 2232 | Sk: paul.wms
 paul.willi...@flatironssolutions.com 
 mailto:paul.willi...@flatironssolutions.com
  
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Error while creating MarkLogic Forest with HDFS as data directory

[Justin Makeig]

Which version of MarkLogic and which vendor and version of Hadoop are you using?

Justin


--
Justin Makeig
Director, Product Management
MarkLogic

 On Mar 13, 2015, at 7:48 AM, Sudheer Yalaverthi sudheer.yalaver...@db.com 
 wrote:
 
  
 Hi,
  
 I have been trying a MarkLogic tiered storage POC with HDFS as storage layer 
 for one of the tier. I have been trying to create a forest with data 
 directory as hadoop file system directory.
 
 I have one hadoop cluster and one Marklogic cluster. I downloaded the 
 configuration files from Hadoop and copied them to /usr/hadoop directory on 
 the MarkLogic machine and I have also downloaded the required jar files based 
 on the documentation here. 
 https://docs.marklogic.com/guide/performance/disk-storage#id_27091 
 https://docs.marklogic.com/guide/performance/disk-storage#id_27091
 I have placed these as well in the /usr/hadoop directory with proper lib 
 structure.
 
 I am getting the below error when I try creating the forest.
 
 2015-03-12 19:17:20.087 Error: Automount Foresthadoop: SVC-HDFSNOT: HDFS not 
 available for 'hdfs://{namdenode_hostname}:8020/tmp': unknown error
 
 I tried changing the log level to finest in the group configurations and I 
 have also added trace events for forest. But I am not able to get any 
 additional details that could point me to what the error is about.
 
 Any help in this regard would be appreciated. Please let me know if there are 
 any other ways to connect to HDFS as a forest directory.
 
  
  
  
 Thanks,
 Sudheer
 
 
 ---
 This communication may contain confidential and/or privileged information. If 
 you are not the intended recipient (or have received this communication in 
 error) please notify the sender immediately and destroy this communication. 
 Any unauthorized copying, disclosure or distribution of the material in this 
 communication is strictly forbidden.
 
 Deutsche Bank does not render legal or tax advice, and the information 
 contained in this communication should not be regarded as such.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] fn:replace Query

[Justin Makeig]

You need to escape the +. That's a special character in regular expressions, 
meaning at least one.

Justin


 On Mar 11, 2015, at 2:25 AM, Kapoor, Pragya pkapo...@innodata.com wrote:
 
 Hi,
 
 I am running the below query
 
 fn:replace(fn:concat('test','-',fn:current-dateTime()),'+','@')
 
 This query is giving the following error:
 [1.0-ml] XDMP-REGEX: (err:FORX0002) 
 fn:replace(test-2015-03-11T14:46:58.44+05:30, +, @) -- Invalid regular 
 expression
 ​
 Please let me know, why ML is behaving this way.
 
 Thanks
 Pragya
  
 This e-mail and any attachments transmitted with it are for the sole use of 
 the intended recipient(s) and may contain confidential , proprietary or 
 privileged information. If you are not the intended recipient, please contact 
 the sender by reply e-mail and destroy all copies of the original message. 
 Any unauthorized review, use, disclosure, dissemination, forwarding, printing 
 or copying of this e-mail or any action taken in reliance on this e-mail is 
 strictly prohibited and may be unlawful.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Finding Top 10 words and collections or directories searching in java

[Justin Makeig]

Typically a huge xml data is an indication that you're using XML documents 
more like relational tables than rows, to draw a comparison. Documents in 
MarkLogic are designed to act more like rows.  (And collections more like 
tables.) Put individual records in their own documents. If you have many 
Product instances, you into its own document. This may require some up-front 
processing to split things out. (See, for example, 
http://docs.marklogic.com/guide/ingestion/content-pump#id_65814.) From there 
you can set range indexes on any or all of a Product's dimensions. Use 
cts:values() or the related lexicon functions to get distinct values and their 
frequencies. 

If you can provide some more details on what your data and queries look like, 
others may be able to help more.

Justin


--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com


 On Feb 13, 2015, at 1:50 PM, Maisnam Ns maisnam...@gmail.com wrote:
 
 Hi,
 
 Can someone help with this use case.
 
 I have a huge xml data in which product is one of the elements. I want to 
 find the top 10 products from these data.
 
 Product is not in the range index and will not be part of facets. 
 How to search this with JAVA API and not with xquery.
 
 Secondly, I need to divide the data into two parts. In marklogic there are 
 directories and collections. 
 
 But how do I search a string from say part A if data is divided into part A 
 and part B.There is an option to select just from part A , part B and both 
 Part A and Part B. Depending on selection of options, if I select Part A , 
 the string has to search from Part A likewise for Part B and if both A and B 
 is selected it has to search from both A and B.
 
 Please let me know how to do this in java. A snippet of code will be highly 
 appreciated.
 
 And , information studio of Marklogic does not provide any option for 
 collections , it only provides for different directories.
 
 Thanks in advance
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general





smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Searching Standalone Properties Documents

[Justin Makeig]

Can you say a little more about what you're trying to accomplish? While it's 
possible to work with stand-alone properties (e.g. 
http://docs.marklogic.com/cts:properties-query), it's not a well trodden 
path. You'd be much better off if you can reformulate your implementation using 
tried and true document fragments.

Justin

 On Jan 19, 2015, at 9:28 AM, neil bradley n...@bradley.co.uk wrote:
 
 Hi, I am having a problem working out how (if it is possible) to use
 cts:search to find specific STANDALONE properties documents
 (properties that are not associated with an actual document).
 
 Every formulation of an XPath that I try either fails to find anything
 that I know it should find, or says its UNREACHABLE.
 
 It does not surprise me that doc() does not work, because there IS no
 document, and that collection() also does not work for a similar
 reason.
 
 I had hoped I could do something like:
 
  /property::*
 
 But that is not working.
 
 So, anyone know if its possible, and if so what kind of XPath I need?
 
 Regards,
 
 Neil.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] function index in MarkLogic?

[Justin Makeig]

As Geert indicates, there is no exactly comparable functionality in MarkLogic 
today. You can think of the document—structure and values—as the API to the 
indexes. Thus, if you want to index something it needs to be (mostly) explicit 
in the document itself. (Word stemming is a counter example of something that's 
automatically derived index time.) I think your best bet will be to compute the 
derived values for inserts or updates and batch process your existing content. 
You can use CPF http://docs.marklogic.com/guide/cpf or lower level triggers 
http://docs.marklogic.com/guide/app-dev/triggers to manage the steady-state 
workflow or orchestrate externally depending on what you have in front of 
MarkLogic (e.g. an application/service layer in Java, for example).

Justin

 On Jan 8, 2015, at 12:19 PM, Geert Josten geert.jos...@marklogic.com wrote:
 
 Hi Alexei,
 
 On the question whether MarkLogic allows indexing function results, then the 
 answer is unfortunately no. Such a feature is on the wish list of many of us.
 
 The best alternative for upper-case would be using collations to ignore case, 
 for instance http://marklogic.com/collation//S1 (ignore case and diacritics) 
 or http://marklogic.com/collation//S2 (ignore case). You can use the 
 collation builder in the admin interface to tweak the collation uri, or 
 search the documentation to find out all options.
 
 Kind regards,
 Geert
 
 From: Alexei Betin abe...@elevate.com
 Reply-To: MarkLogic Developer Discussion general@developer.marklogic.com
 Date: Thursday, January 8, 2015 at 8:47 PM
 To: general@developer.marklogic.com general@developer.marklogic.com
 Subject: [MarkLogic Dev General] function index in MarkLogic?
 
 Hello,
 
  
 
 Does MarkLogic has something similar to “function index” as exists in other 
 databases such as Oracle? For example, in Oraclecreate a function index on a 
 value of UPPER(field_name) as follows:
 
 CREATE INDEX cities_fn_idx ON cities (UPPER(name));
 
 which ensures that a query like the one below is efficient and not using the 
 full table scan:
 
 SELECT name FROM cities WHERE UPPER(name) = 'HOUSTON';
 
  
 
 I’d like to do something similar with MarkLogic which could help speed up, 
 e.g. the following simple query (which currently prohibitively slow despite 
 an index on /A/City):
 
  
 
 collection()/A/[upper-case( City ) = 'HOUSTON']
 
  
 
 upper-case() is only an example, I am looking for a solution that would 
 accommodate any function/transformation that to be used in a query condition 
 from an existing element.
 
  
 
 Clearly, one solution would be to pre-compute the derived field, add it to 
 all the documents in the collection, and create an a “function index”).
 
  
 
 Thanks,
 
  
 
 image005.jpg
 image006.jpg
 Alexei Betin
 
 Principal Architect; Big Data
 P: (817) 928-1643 | Elevate.com
 4150 International Plaza, Suite 300
 Fort Worth, TX 76109
 
  
 
 Privileged and Confidential. This e-mail, and any attachments thereto, is 
 intended only for use by the addressee(s) named herein and may contain 
 privilegedthereto, is strictly prohibited.
 
  
 
 image005.jpgimage006.jpg___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Hadoop MarkLogic data migration

[Justin Makeig]

Yes, mlcp http://developer.marklogic.com/products/mlcp can move data between 
HDFS and MarkLogic. The Connector for Hadoop  
http://developer.marklogic.com/products/hadoop can also run Map Reduce jobs 
with MarkLogic as an Input or OutputFormat. Bring data from Hadoop can mean 
many things. If you can describe they type, format, and volume of data we can 
probably better target a specific solution.

Justin


--
Justin Makeig
Director, Product Management
MarkLogic
justin.mak...@marklogic.com

On Nov 11, 2014, at 8:53 PM, abhishek.srivas...@cognizant.com wrote:

 Hi All,
 
 I want to bring data from Hadoop into MarkLogic cluster. Is there any tool 
 available to move data from Hadoop to ML?
 
 Thanks
 Abhishek
 This e-mail and any files transmitted with it are for the sole use of the 
 intended recipient(s) and may contain confidential and privileged 
 information. If you are not the intended recipient(s), please reply to the 
 sender and destroy all copies of the original message. Any unauthorized 
 review, use, disclosure, dissemination, forwarding, printing or copying of 
 this email, and/or any action taken in reliance on the contents of this 
 e-mail is strictly prohibited and may be unlawful. Where permitted by 
 applicable law, this e-mail and other e-mail communications sent to and from 
 Cognizant e-mail addresses may be monitored.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general





smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] What's a best approach to search unstructured docs for numbers Greater Than N?

[Justin Makeig]

Try creating a path range index on a reasonably expansive path, e.g. 
/doc/(prices|cost)/*. Then you can query like any other range index. You can 
use cts:valid-index-path(/doc/(prices|cost)/*, false()) to validate that your 
path can be used in a range index.

Justin

On Oct 28, 2014, at 4:15 PM, Gary Russo garyru...@hotmail.com wrote:

 I have a database that consists of highly unstructured documents. The 
 documents contain pricing data.
  
 My requirement is to search all nodes of every document to find a number that 
 is greater than 1,000.
  
 The element names containing the numbers can be anything.
  
 Example Docs:
  
 Doc 1
  
 doc
   prices
 beginningBalance250/beginningBalance
 endingBalance9996.26426957374/endingBalance
   /prices
   summary
 summary-asPublic Law/summary-as
 summary-date2012-02-14/summary-date
 summary-textRedesignates the Noxubee National Wildlife 
 Refuge./summary-text
   /summary
   bankCharge1500.75/bankCharge
 /doc
 
  
 Doc 2
  
 doc
   cost
 startBalance250/startBalance
 endBalance9996.26426957374/endBalance
   /cost
   summary
 summary-asPublic Law/summary-as
 summary-date2012-02-14/summary-date
   /summary
   bankFee1500.75/bankFee
 /doc
 
  
 I can use a brute force XQuery code snippet like the following but I’d like 
 to use the universal index.
  
 What is the recommended approach for something like this?
  
  
 Brute Force XQuery:
  
 let $values :=
   for $n in $doc1//node()/*/text()
 let $value := try { xs:float($n) } catch ($e) { () }
 where $value gt 1000
   return
 $value|| | ||xdmp:path($n)
 return $values
  
  
  
  
 Gary Russo
 Enterprise NoSQL Developer
 http://garyrusso.wordpress.com
  
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] How to optimize the REST API Bulk Ingestion Performance?

[Justin Makeig]

The bulk API does not spawn tasks in the Server. It allows you to send groups 
of documents together in a single request. (The implementation is actually HTTP 
multipart.) An E-node can, of course, handle multiple simultaneous requests, 
bulk or not.

Why spawn in the Server? Without knowing much more about what the bottlenecks 
actually are, I'd start by spawning threads in C#, similar the way that 
something like mlcp or Corb does. It sounds like you have some headroom in 
MarkLogic so you should be able to throw more work at it from the client. 

Justin


On Oct 14, 2014, at 12:16 PM, Gary Russo garyru...@hotmail.com wrote:

 Hello Danny,
  
 Yes, I’m using 7.0-4.
  
  What are you comparing it to on the Oracle side?
  In MarkLogic, the content will be all indexed and searchable.  Is that 
  true on the orcl side too
  
 The Oracle side is doing a basic CLOB insert with no indexing.
  
 The Oracle server being compared to is a higher capacity system so we 
 expected to see a faster ingestion.
  
 I didn’t expect the MarkLogic side to be 4 times slower.
  
 Yes, we tried tweaking the batch size. The 500 batch size had the fastest 
 load times.
  
 I will investigate further but I believe the bottleneck is on the MarkLogic 
 side.
  
 I believe the MarkLogic CPU has some room for parallelizing.
  
 I’ll create a custom REST Extension that will spawn multiple threads for the 
 doc-inserts.
  
 I assume the REST API bulk ingestion already does this but I can’t say for 
 sure.
  
 I’ll keep you posted.
  
 Thanks Danny
  
 -  Gary R
  
  
  
 From: general-boun...@developer.marklogic.com 
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of Danny Sokolsky
 Sent: Tuesday, October 14, 2014 2:00 PM
 To: MarkLogic Developer Discussion
 Subject: Re: [MarkLogic Dev General] How to optimize the REST API Bulk 
 Ingestion Performance?
  
 Hi Gary,
  
 A few thoughts here.  You are using 7.0-4 on this? 
  
 What are you comparing it to on the Oracle side?  In MarkLogic, the content 
 will be all indexed and searchable.  Is that true on the orcl side too?
  
 What indexes to you have enabled?  Maybe you do not need them all (or maybe 
 you should put the equivalent indexing on the orcl side)?
  
 Have you tried tweaking the batch size?  I would try a smaller number, say 50 
 or 100.
  
 Have you analyzed where you are spending the time?  In the c# code?  In the 
 code loading the doc on MarkLogic?
  
 Do you have multiple threads loading from your .net program?  If you are not 
 maxing out your cpu on the MarkLogic side, you probably have room for more 
 parallelization.
  
 -Danny
  
 From: general-boun...@developer.marklogic.com 
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of Gary Russo
 Sent: Tuesday, October 14, 2014 9:21 AM
 To: general@developer.marklogic.com
 Subject: [MarkLogic Dev General] How to optimize the REST API Bulk Ingestion 
 Performance?
  
 MarkLogic Bulk ingestion processing is slower than an equivalent Oracle 
 ingestion process.
  
 The MarkLogic ingestion takes 30 minutes. An Oracle equivalent only takes 7 
 minutes.
  
 I’m using the REST API to bulk ingest multiple documents as described here. 
 = http://docs.marklogic.com/guide/rest-dev/bulk#id_54649
  
 Notes:
 · C# code is used to call the MarkLogic Bulk Ingest REST API.
 · Document batch size used is 500.
 · Average doc size is 1 KB.
 · JSON Conversion and Validation logic occurs in the C# code.
  
  
 Any thoughts on how to optimize the MarkLogic bulk ingest to make it as fast 
 as Oracle’s 7 minute load time?
  
  
 Thanks,
 Gary R
  
  
 Gary Russo
 Enterprise NoSQL Developer
 http://garyrusso.wordpress.com
  
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Use different directory for full text search

[Justin Makeig]

Not knowing anything else about your app, this sounds like a good use case for 
the “envelope” pattern. Rather than  separate article…/article and 
descriptor…/descriptor documents, you can combine them into something like

ns:envelope
  descriptor…/descriptor
  article…/article
/ns:envelope

You can use term weights to tune relevance based on which part of the document 
a search match comes from. Having all of the necessary information in a single 
fragment means you won't get duplicate results, won't have to do joins reunite 
the descriptor with its article, and you'll only have to update one document 
for changes.

Going in this direction, of course, will require a one-time bulk 
transformation. 

Justin


--
Justin Makeig
Director, Product Management
MarkLogic
jmak...@marklogic.com

On Oct 2, 2014, at 3:11 PM, Danny Sinang d.sin...@gmail.com wrote:

 Hi,
 
 We have article documents, each of which has a corresponding descriptor 
 document that contains key elements from the its parent article, but 
 structured differently for easier searching.
 
 All the articles are stored under the articles directory and all the 
 descriptor docs in the descriptors directory.
 
 My search:search code limits its searches to the descriptor folder by 
 specifying this option :
 
  additional-query xmlns=http://marklogic.com/appservices/search;
 { cts:directory-query(/descriptors, infinity) }
   /additional-query
 
 
 All was fine until I ran into a requirement to support full text searches on 
 the body and citation elements - both of which we don't and can't have in 
 the descriptor documents (not wanting its size to blow up).
 
 I'm thinking of making search:search look inside either /articles or 
 /descriptors, but then I realized, given the similar content between articles 
 and descriptors, I can/will get duplicate results.
 
 An alternative would be to limit my searches to the /articles directory for 
 ALL kind of queries but then I'd have to modify all my custom constraints to 
 search through different elements inside article documents.
 
 The last option I have in mind is to make search:search look inside /articles 
 instead of /descriptors for ALL full text searches only. Is this possible ? 
 How ?
 
 Perhaps there is a better way around this that I haven't thought of. Would 
 appreciate any suggestions.
 
 Thanks in advance.
 Danny
 
 
 
 
 
 
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general




smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Number of fragments per document

[Justin Makeig]

Yes, properties, (manually created) directories, and locks are stored in 
fragments. You also might be counting deleted fragments, i.e. those marked for 
deletion but not yet merged out, depending on how you’re counting (see 
http://docs.marklogic.com/xdmp:forest-counts). Take a look at 
http://developer.marklogic.com/blog/fragmented-thoughts for more information 
on fragments.

Justin

--
Justin Makeig
Director, Product Management
MarkLogic

On Sep 4, 2014, at 3:47 PM, Danny Sinang d.sin...@gmail.com wrote:

 Hi,
 
 I noticed there are more fragments than documents in my ML databases which 
 have no Fragment Roots or Fragment Parents configured.
 
 Is this by design ?
 
 Are document properties stored in another fragment ? If so, then there should 
 be twice the number of fragments than documents. But how come I see 2.6x 
 fragments than my documents ?
 
 
 Regards,
 Danny
 
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general




___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] WebDAV app service code available or other implementation for WebDAV to MarkLogic?

[Justin Makeig]

No, that's not possible. MarkLogic's WebDAV app server is implemented in C++, 
deep in the guts of the Server. 
What types of customization are you looking to do? Is WebDAV a hard and fast 
requirement or would a custom REST service suffice?
As was discussed on a recent thread, you could stick a reverse HTTP proxy in 
between your WebDAV client and MarkLogic and do processing there. You could 
also implement your own WebDAV server in XQuery with an HTTP app server. If 
WebDAV is not a hard and fast requirement, you could also create your own 
custom (likely simpler) REST services or extend the ones that are built into 
MarkLogic http://docs.marklogic.com/guide/rest-dev/intro#chapter.

Justin

Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com



On Jul 9, 2014, at 9:24 AM, Casey Jordan casey.jor...@jorsek.com wrote:

 If someone wanted to customize the WebDAV app service does Mark Logic make 
 this code available so that it could be modified and run on a separate 
 application server (Ie inside a servlet container)?
 
 -- 
 --
 Casey Jordan
 easyDITA a product of Jorsek LLC
 CaseyDJordan on LinkedIn, Twitter  Facebook
 (585) 348 7399
 easydita.com
 
 
 This message is intended only for the use of the Addressee(s) and may
 contain information that is privileged, confidential, and/or exempt from
 disclosure under applicable law.  If you are not the intended recipient,
 please be advised that any disclosure  copying, distribution, or use of
 the information contained herein is prohibited.  If you have received
 this communication in error, please destroy all copies of the message,
 whether in electronic or hard copy format, as well as attachments, and
 immediately contact the sender by replying to this e-mail or by phone.
 Thank you.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Custom serializer for WebDAV or other Services

[Justin Makeig]

No, not for WebDAV. With XQuery or XDBC you have full control. What, in 
particular, are you trying to customize?

Justin

Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com



On Jul 8, 2014, at 1:08 PM, Casey Jordan casey.jor...@jorsek.com wrote:

 Is there a way to have a custom serializer for content that is serialized out 
 of the database but just for specific connections? 
 
 For instance, so that XML served through WebDAV could be serialized 
 differently than when just retrieved and displayed through XQuery?
 
 Thanks
 
 
 
 -- 
 --
 Casey Jordan
 easyDITA a product of Jorsek LLC
 CaseyDJordan on LinkedIn, Twitter  Facebook
 (585) 348 7399
 easydita.com
 
 
 This message is intended only for the use of the Addressee(s) and may
 contain information that is privileged, confidential, and/or exempt from
 disclosure under applicable law.  If you are not the intended recipient,
 please be advised that any disclosure  copying, distribution, or use of
 the information contained herein is prohibited.  If you have received
 this communication in error, please destroy all copies of the message,
 whether in electronic or hard copy format, as well as attachments, and
 immediately contact the sender by replying to this e-mail or by phone.
 Thank you.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Custom serializer for WebDAV or other Services

[Justin Makeig]

That should read with HTTP or XDBC [app servers] you have full control. And 
that control comes from running custom XQuery.

Justin


On Jul 8, 2014, at 1:11 PM, Justin Makeig justin.mak...@marklogic.com wrote:

 No, not for WebDAV. With XQuery or XDBC you have full control. What, in 
 particular, are you trying to customize?
 
 Justin
 
 Justin Makeig
 Director, Product Management
 MarkLogic Corporation
 justin.mak...@marklogic.com
 www.marklogic.com
 
 
 
 On Jul 8, 2014, at 1:08 PM, Casey Jordan casey.jor...@jorsek.com wrote:
 
 Is there a way to have a custom serializer for content that is serialized 
 out of the database but just for specific connections? 
 
 For instance, so that XML served through WebDAV could be serialized 
 differently than when just retrieved and displayed through XQuery?
 
 Thanks
 
 
 
 -- 
 --
 Casey Jordan
 easyDITA a product of Jorsek LLC
 CaseyDJordan on LinkedIn, Twitter  Facebook
 (585) 348 7399
 easydita.com
 
 
 This message is intended only for the use of the Addressee(s) and may
 contain information that is privileged, confidential, and/or exempt from
 disclosure under applicable law.  If you are not the intended recipient,
 please be advised that any disclosure  copying, distribution, or use of
 the information contained herein is prohibited.  If you have received
 this communication in error, please destroy all copies of the message,
 whether in electronic or hard copy format, as well as attachments, and
 immediately contact the sender by replying to this e-mail or by phone.
 Thank you.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Custom serializer for WebDAV or other Services

[Justin Makeig]

Yes, if you need WebDAV. The REST and Java Client APIs also allow for custom 
outbound (and inbound) transformations. (That doesn't get you closer to WebDAV, 
though.)

Justin


On Jul 8, 2014, at 1:13 PM, Casey Jordan casey.jor...@jorsek.com wrote:

 Well I need to strip out particular attributes depending on where content is 
 being served from. For WebDAV and REST I don't see an easy way to do this. 
 
 Would this mean that I would need to implement my own WebDAV client, so I 
 could inject this logic?
 
 
 On Tue, Jul 8, 2014 at 4:11 PM, Justin Makeig justin.mak...@marklogic.com 
 wrote:
 No, not for WebDAV. With XQuery or XDBC you have full control. What, in 
 particular, are you trying to customize?
 
 Justin
 
 Justin Makeig
 Director, Product Management
 MarkLogic Corporation
 justin.mak...@marklogic.com
 www.marklogic.com
 
 
 
 On Jul 8, 2014, at 1:08 PM, Casey Jordan casey.jor...@jorsek.com wrote:
 
 Is there a way to have a custom serializer for content that is serialized 
 out of the database but just for specific connections? 
 
 For instance, so that XML served through WebDAV could be serialized 
 differently than when just retrieved and displayed through XQuery?
 
 Thanks
 
 
 
 -- 
 --
 Casey Jordan
 easyDITA a product of Jorsek LLC
 CaseyDJordan on LinkedIn, Twitter  Facebook
 (585) 348 7399
 easydita.com
 
 
 This message is intended only for the use of the Addressee(s) and may
 contain information that is privileged, confidential, and/or exempt from
 disclosure under applicable law.  If you are not the intended recipient,
 please be advised that any disclosure  copying, distribution, or use of
 the information contained herein is prohibited.  If you have received
 this communication in error, please destroy all copies of the message,
 whether in electronic or hard copy format, as well as attachments, and
 immediately contact the sender by replying to this e-mail or by phone.
 Thank you.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 
 
 
 -- 
 --
 Casey Jordan
 easyDITA a product of Jorsek LLC
 CaseyDJordan on LinkedIn, Twitter  Facebook
 (585) 348 7399
 easydita.com
 
 
 This message is intended only for the use of the Addressee(s) and may
 contain information that is privileged, confidential, and/or exempt from
 disclosure under applicable law.  If you are not the intended recipient,
 please be advised that any disclosure  copying, distribution, or use of
 the information contained herein is prohibited.  If you have received
 this communication in error, please destroy all copies of the message,
 whether in electronic or hard copy format, as well as attachments, and
 immediately contact the sender by replying to this e-mail or by phone.
 Thank you.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Guides on database design for multi-tenancy?

[Justin Makeig]

Casey,
There are two ways in MarkLogic 7 to query a specific database: Create a 
separate app server (HTTP or XDBC) for each database. An app server has a 
default database that you can set in configuration. Each query/update evaluated 
for that app server runs against that database. Many app servers can point to 
one database, but an app server can only be associated with one database. 
Another, lower-level means is to use xdmp:eval 
http://docs.marklogic.com/xdmp:eval?q=xdmp:eval or xdmp:invoke. These allow 
you to specify a database at runtime and evaluate specific code against it. I 
wouldn't recommend this as a general approach, though. It will make your code 
less readable and, in certain scenarios, will prevent MarkLogic from maximizing 
some performance optimizations it does under the covers.

Another approach might be to create protected collections for each tenant 
within the same database. With MarkLogic's role-based security, you can be 
assured that you can completely restrict viewing and editing to very specific 
roles. You can take a similar approach to running privileged code with amps. 
Take a look at the Security Guide for more details 
http://docs.marklogic.com/guide/admin/security#chapter.

Justin



Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com



On Jul 7, 2014, at 10:14 AM, Casey Jordan casey.jor...@jorsek.com wrote:

 Hi all,
 
 I am checking out Mark Logic for the first time and I was interested if there 
 is any information around designing a cluster for multi-tenancy?
 
 I assumed that I could create a separate database for each client that 
 would be using the application, and then segment data that way. However right 
 away it became a little unclear to me as to how I query a specific database 
 (couldn't find an example of this in the docs), or manage users, triggers, 
 schemas etc for a specific database. 
 
 I know this is a fairly general question, but any advice would be helpful.
 
 Thanks
 
 -- 
 --
 Casey Jordan
 easyDITA a product of Jorsek LLC
 CaseyDJordan on LinkedIn, Twitter  Facebook
 (585) 348 7399
 easydita.com
 
 
 This message is intended only for the use of the Addressee(s) and may
 contain information that is privileged, confidential, and/or exempt from
 disclosure under applicable law.  If you are not the intended recipient,
 please be advised that any disclosure  copying, distribution, or use of
 the information contained herein is prohibited.  If you have received
 this communication in error, please destroy all copies of the message,
 whether in electronic or hard copy format, as well as attachments, and
 immediately contact the sender by replying to this e-mail or by phone.
 Thank you.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] MLCP issue

[Justin Makeig]

Can you show the mlcp command line arguments that you used? The version of 
mlcp, MarkLogic, and Java would also help triangulate the problem.

Justin

Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com

On Jun 6, 2014, at 9:23 AM, abhishek.srivas...@cognizant.com 
abhishek.srivas...@cognizant.com wrote:

 Hi All
 
 While loading data from MLCP I am geeting following exception
 
 Default provider not a usable net address.
 
 I tried pinging MarkLogic server and they are accessible.
 
 Please suggest
 
 Thanks
 Abhishek
 
 This e-mail and any files transmitted with it are for the sole use of the 
 intended recipient(s) and may contain confidential and privileged 
 information. If you are not the intended recipient(s), please reply to the 
 sender and destroy all copies of the original message. Any unauthorized 
 review, use, disclosure, dissemination, forwarding, printing or copying of 
 this email, and/or any action taken in reliance on the contents of this 
 e-mail is strictly prohibited and may be unlawful. Where permitted by 
 applicable law, this e-mail and other e-mail communications sent to and from 
 Cognizant e-mail addresses may be monitored.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general








smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] YAML support

[Justin Makeig]

MarkLogic does not ingest YAML natively. However, doing a client-side 
conversion to XML before inserting should be pretty straightforward and 
lossless. What does your current (or planned) ingestion process look like? With 
a little more information about your environment, someone might be able to 
suggest a specific conversion strategy.

Justin


On May 6, 2014, at 6:56 AM, Mohamad Thahir thahirmoha...@gmail.com wrote:

 Does MarkLogic support YAML for ingestion? Can someone recommend best way to 
 ingest YAML docs 
 
 Sent from my iPhone
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com





smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Is there a performance penaly for using long directory and document uris?

[Justin Makeig]

Theoretically, I guess. But I'd expect that would be infinitesimal compared to 
other things that your code is doing. The usability of URI structures for your 
developers is what I'd suggest optimizing on. (And don't use directory 
fragments if you're worried about performance.)

Justin


Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com


On Apr 4, 2014, at 11:40 AM, Tim t...@aaom.net wrote:

 Sounds like that is a limit only if using WebDAV. Aside from WebDAV are there
 any performance issues?
 
 -Original Message-
 From: general-boun...@developer.marklogic.com
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of Will Thompson
 Sent: Friday, April 04, 2014 2:27 PM
 To: MarkLogic Developer Discussion
 Subject: Re: [MarkLogic Dev General] Is there a performance penaly for using
 long directory and document uris?
 
 I think the short answer is yes due to lock contention on the directory
 fragments. I think this can be mitigated somewhat by manually creating the
 directory fragments rather than turning on automatic directory creation. For
 detailed information on this topic, I always refer to Mike's blog post:
 http://blakeley.com/blogofile/2012/03/19/directory-assistance/
 
 -Will
 
 On Apr 4, 2014, at 1:13 PM, Tim t...@aaom.net wrote:
 
 Hi Folks,
 
 A long directory URI may be useful for facilitating webDAV access to content
 or simply to organize content by a virtual file system, for example,
 
  /category/subcategory/in-work/year/docname.xml
 
 which could as easily have been stored as 
 
  /docname.xml
 
 Is there a performance penalty having a longer document URI?
 
 Thanks again,
 
 Tim Meagher
 
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general





___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Post-read trigger?

[Justin Makeig]

Thanks, Mike.
MarkLogic Product Management does actively monitor this list.
Geert, if I’m reading your requirements correctly, you’d like provide “advice” 
(in the AOP sense) around read access to documents in order to count 
invocations. If you literally want to do this for the fn:doc function, Mike’s 
solution with amps would work pretty cleanly today. This would require 
consumers to know about the josten:doc() function, though. The read 
trigger/hook is interesting, but as Mike indicates, many other capabilities in 
the Server “read” documents. Would a lexicon access count as a view? The more 
general use case for before, after, and around advice is an interesting one.

Justin

Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com



On Feb 4, 2014, at 11:13 AM, Michael Blakeley m...@blakeley.com
 wrote:

 I'm not sure how closely product management monitors this list. It might be 
 more efficient to send RFEs through support channels.
 
 That aside, it sounds like you're looking for a database-level way to 
 discriminate between search summary results and document views? I think 
 that's best done in application code, because the application code defines 
 the difference between those two types of database reads.
 
 If you want to make sure the view-count isn't circumvented by non-admin 
 users, consider using document permissions and amps. You could withhold 
 document-read access from non-admin users. Instead they would have to call 
 application-specific amp'd functions. Those amp'd functions would take care 
 of all document search and document access. In the case of document access - 
 as defined by your application logic - the amp'd function would also update 
 the view count.
 
 BTW if I were writing an RFE for your use-case, I would call it a 
 function-specific hook rather than a trigger. Such a feature might allow 
 registration of arbitrary code hooks into arbitrary function calls, or at 
 least a list of important ones. Besides your use-case, that could be useful 
 for debugging and profiling. Registering those hooks might look something 
 like the system plugin framework: 
 http://docs.marklogic.com/guide/app-dev/plugins
 
 On the other hand if engineering decided to implement a database read trigger 
 I would expect a generic one, firing on any document read whatsoever. 
 Triggers are low-level, so they have to target low-level database 
 functionality: update, delete, etc.
 
 That would probably disappoint you, because it would not be restricted to 
 fn:doc only. Instead I would expect a trigger on database reads to fire on 
 fn:doc, and also on cts:search, fn:collection, xdmp:directory, some XPath 
 expressions, and maybe property-read functions. It would probably fire once 
 per document read. I know that isn't what *you* want right now, but that's 
 what a high-security application would want. Otherwise a hostile user could 
 still use cts:search and cts:document-query to retrieve documents by URI, or 
 fall back on fn:collection, xdmp:directory, or XPath expressions. Engineering 
 would probably try to cover both use-cases.
 
 So using a read trigger to track fn:doc calls might get ugly, simply because 
 you would have to ignore most of the trigger evaluations. Depending on the 
 implementation you might need clever trigger code to discriminate between all 
 these different kinds of reads, and only increment your view-count for the 
 real document views. Given all this, I think a function-specific hook would 
 be more practical.
 
 -- Mike
 
 On 3 Feb 2014, at 23:29 , Geert J. geert.jos...@dayon.nl wrote:
 
 Hi,
 
 Would a post-read trigger (only at accessing a document explicitly using
 doc()) make sense? For instance to update a view-count in the document
 properties. Saves one from spawning a task oneself..
 
 Cheers
 
 M.Sc. G.P.H. (Geert) Josten
 Senior Developer
 
 
 Dayon B.V.
 Delftechpark 37b
 2628 XJ Delft
 The Netherlands
 
 T +31 (0)88 26 82 570
 
 geert.jos...@dayon.nl
 www.dayon.nl
 
 De informatie - verzonden in of met dit e-mailbericht - is afkomstig van
 Dayon BV en is uitsluitend bestemd voor de geadresseerde. Indien u dit
 bericht onbedoeld hebt ontvangen, verzoeken wij u het te verwijderen. Aan
 dit bericht kunnen geen rechten worden ontleend.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Stemming constraint

[Justin Makeig]

cts:stem(…) http://docs.marklogic.com/cts:stem will tell you how the server 
is stemming words. You can also customize which words are mapped to which stems 
using a custom dictionary 
http://docs.marklogic.com/guide/search-dev/custom-dictionaries. Make sure you 
heed the caution in the docs about reindexing if you go down the custom 
dictionary path.

Justin



Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.commailto:justin.mak...@marklogic.com
www.marklogic.comhttp://www.marklogic.com/



On Jan 16, 2014, at 11:38 PM, 
diwakar.narayanasw...@cognizant.commailto:diwakar.narayanasw...@cognizant.com
 wrote:

Hi all,
I have a document which has a word “hearings”. I tried 
searching the document using cts:search with that word. I got the document. 
Once again I tried searching the a same document by giving the word “hearing”. 
The corresponding document with the word “hearings” did not come in the result.

So I tried
Cts:stem(hearing) – I did not get  “ hearings”.


Doesn’t “hearings” and “hearing” come from the same root. Is this a Mark logic 
limitation or English language constraint?

Thanks in Advance,
Diwakar N
This e-mail and any files transmitted with it are for the sole use of the 
intended recipient(s) and may contain confidential and privileged information. 
If you are not the intended recipient(s), please reply to the sender and 
destroy all copies of the original message. Any unauthorized review, use, 
disclosure, dissemination, forwarding, printing or copying of this email, 
and/or any action taken in reliance on the contents of this e-mail is strictly 
prohibited and may be unlawful. ___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] mlcp error

[Justin Makeig]

Jakob,
Sorry about the unresponsiveness. That looks like you might have uncovered a 
bug. I’ll have an Engineer on our end try to reproduce this and contact you 
offline for more details. I’ll follow up with the list with the eventual 
outcome. Thanks for your feedback.

Justin



Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.commailto:justin.mak...@marklogic.com
www.marklogic.comhttp://www.marklogic.com/



On Nov 6, 2013, at 7:12 AM, Jakob Fix 
jakob@gmail.commailto:jakob@gmail.com
 wrote:

Hello,

I was just wondering whether the silence in response to my question is 
stunned, not interested, well, can't he figure this out by himself or 
oops, didn't see this one?

:-)

cheers,
Jakob.


On Fri, Nov 1, 2013 at 12:01 AM, Jakob Fix 
jakob@gmail.commailto:jakob@gmail.com wrote:
Hi,

we've run into something we think might be a bug with the most recent version 
of mlcp. We did an export of a database with XML documents and lots of binary 
documents, and an import of the exported data into another database.  In the 
second step of the procedure, i.e. the import into the new database, the error 
below appeared (the line with Archive damaged ...). Apparently, mlcp stores XML 
and binary documents in different zip files. Also, each binary document gets 
its metadata document. In our case, the export created two zip files containing 
the binaries. For some reason, in the case of one document, the actual binary 
file and its metadata file were separated, as shown below:

20131031140432+0100-01-BINARY.zip == RO-GE_DTC.pdf.metadata
20131031140432+0100-02-BINARY.zip == RO-GE_DTC.pdf

which seems to have caused the error below. The PDF file is indeed not loaded 
into the database.

Reuniting the PDF file with its metadata equivalent in the same binary zip file 
made the import procedure run without errors.

thanks,
Jakob.


marklogic-contentpump-1.0.3\bin\mlcp.bat EXPORT -host 192.168.56.90 -port 5 
-username abc  -password abc  -output_type archive -output_file_path 
db-prod-20131031

marklogic-contentpump-1.0.3\bin\mlcp.bat IMPORT -host 192.168.56.90 -port 40100 
-username abc -password abc -input_file_path db-prod-20131031 -input_file_type 
archive

13/10/31 14:09:54 INFO contentpump.LocalJobRunner: Content type: XML
13/10/31 14:09:54 INFO jvm.JvmMetrics: Initializing JVM Metrics with 
processName=JobTracker, sessionId=
13/10/31 14:09:54 INFO input.FileInputFormat: Total input paths to process : 3
13/10/31 14:09:55 ERROR contentpump.ArchiveRecordReader: Archive damaged: 
no/incorrect metadata for /content/assets/agreements/RO-GE_DTC.pdf in 
/D:/Projects/EOI/deployment/mlcp/eoi-db-prod-20131032/20131031140432+0100-02-BINARY.zip
13/10/31 14:09:55 ERROR contentpump.LocalJobRunner: Error running task: 
attempt___m_01_0
java.lang.NullPointerException
at 
com.marklogic.contentpump.DatabaseContentWriter.write(DatabaseContentWriter.java:231)
at 
com.marklogic.contentpump.DatabaseContentWriter.write(DatabaseContentWriter.java:58)
at 
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at com.marklogic.contentpump.DocumentMapper.map(DocumentMapper.java:46)
at com.marklogic.contentpump.DocumentMapper.map(DocumentMapper.java:32)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at 
com.marklogic.contentpump.LocalJobRunner$LocalMapTask.call(LocalJobRunner.java:375)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
13/10/31 14:09:56 INFO contentpump.LocalJobRunner:  completed 0%
13/10/31 14:14:42 INFO contentpump.LocalJobRunner:  completed 33%
13/10/31 14:15:41 WARN contentpump.DatabaseContentWriter: SEC-PERMDENIED: 
Permission denied
13/10/31 14:18:27 INFO contentpump.LocalJobRunner:  completed 66%
13/10/31 14:18:27 INFO contentpump.LocalJobRunner: 
com.marklogic.contentpump.ContentPumpStats:
13/10/31 14:18:27 INFO contentpump.LocalJobRunner: 
ATTEMPTED_INPUT_RECORD_COUNT: 20230
13/10/31 14:18:27 INFO contentpump.LocalJobRunner: SKIPPED_INPUT_RECORD_COUNT: 0
13/10/31 14:18:27 INFO contentpump.LocalJobRunner: Total execution time: 512 sec




___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Fast (indexed) query for getting unique absolute Xpaths from each file

[Justin Makeig]

There are techniques you can use to break this work up into smaller chunks, 
using xdmp:spawn http://docs.marklogic.com/xdmp:spawn?q=xdmp:spawn, for 
example. You should also take a look at xdmp:path 
http://docs.marklogic.com/xdmp:path?q=xdmp:path. However, before we dig too 
deep, I’d be interested in knowing what your end goal is. Are you trying to 
summarize the document structures in your database? Look for outliers? Do some 
sort of validation? Something else? The specific use case will likely influence 
the particular solution. Any details would be much appreciated.

Justin


Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.commailto:justin.mak...@marklogic.com
www.marklogic.comhttp://www.marklogic.com/



On Jul 26, 2013, at 2:52 AM, Vaitkus, Arunas (LNG-LON) 
arunas.vait...@lexisnexis.co.ukmailto:arunas.vait...@lexisnexis.co.uk
 wrote:

Dear Marklogic Users,

I have a real problem and got stuck at a dead-end (or so I think at the 
moment!). I have a number of XML files uploaded to MarkLogic server and need to 
run a report listing all possible absolute XPATHs. What this means is that I 
need to parse each XML document, find each node’s unique absolute XPATH (for 
example “/full/xpath/namespace:to/test:specific/node” as I don’t need to know 
about their position using predicates) and then insert a report to another XML 
file.

It is fine with a small set of content (less than 2000 files) as I hit expanded 
tree cache. However, in my case there are tens of thousands of files that make 
everything very slow.

What I am looking for is if there is any indexing capabilities that could help 
me out in this case (getting these XPATHs)?

Currently, my Xquery is very simple and does not reuse any of indexing (as I am 
not sure what I could change to get indexing gains here):

---
xquery version 1.0-ml;
declare namespace html = http://www.w3.org/1999/xhtml;;

report
{
 for $collection in subsequence(collection('0NF9'), 1, 2)
 for $document in $collection
for $node in $document//*
   let $full-xpath := $node/string-join(ancestor-or-self::*/name(), 
'/')  (: this is building textual full XPath representation :)
   let $row := fn:doc('/example.xml')//xpath
  return if (not($row=$full-xpath)) then 
(xdmp:node-insert-child(doc(/example.xml)/report, (xpath 
date={format-date(fn:current-date(), [Y0001]-[M01]-[D01])} 
approved=no{$full-xpath}/xpath))) else ()

}
/report
--

Many Thanks,
Arunas



LexisNexis is a trading name of REED ELSEVIER (UK) LIMITED - Registered office 
- 1-3 STRAND, LONDON WC2N 5JR
Registered in England - Company No. 02746621

___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] searchAPI - authentication

[Justin Makeig]

If you’re really trying to create a search API accessible to a Java client, I’d 
advise you to start with the Java API 
http://developer.marklogic.com/learn/java that we’ve built. Under the covers 
it uses a REST service that ships with the Server, but the API itself is pure 
Java. It allows you to do searches, facets, aggregates, and much more. We’ve 
also put in a lot of effort to make sure it performs well and it is fully 
supported as part of the product.

In your particular case, a 404 indicates that the URL you’ve provided doesn’t 
map to a module path, not an authentication failure. Check your root setting in 
the app server config.

Justin


Justin Makeig
Director, Product Management
MarkLogic Corporation
www.marklogic.comhttp://www.marklogic.com/



On Jul 16, 2013, at 11:49 PM, sini narayanan 
sini.isonl...@gmail.commailto:sini.isonl...@gmail.com
 wrote:

Hi All,

I have a searchAPI created in MarkLogic with the following URL
http://localhost:8008/searchAPI.xqy?name=abcformat=xml

I need to access this page from an external source (JAVA code).
But whenever I invoke the API, I get 404 Page not Found Error.
This is because of the authentication on the searchAPI. I need to provide 
username and password to access the API from JAVA.
How do I achieve this?
Or How do I remove the authentication on the searchAPI page.

Thanks,
Sini
___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

[MarkLogic Dev General] Query Console key bindings

[Justin Makeig]

We’ve had reports from users of non-English keyboard layouts that some of the 
key bindings in Query Console interfere with normal typing. For example, trying 
to type the ever-important “[” character using Alt+5 in a French keyboard 
layout triggers the shortcut to jump directly to the fifth query tab instead. 
We’re proposing eliminating the Alt+# and Ctl+Alt+# key combinations in favor 
of the existing, Ctl+Alt+Shift+Left/Right Arrow combination to move to the 
previous/next tab, similar to the way browser tabs work. Would this 
significantly impact anyone’s workflow? What are the key combinations that you 
use most frequently? (All of the key combinations are documented at 
http://docs.marklogic.com/guide/qconsole/shortcuts.) Feedback on- or off-list 
is appreciated.

Justin


Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com



___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] mlcp option for tranforming documents during export?

[Justin Makeig]

David,
Nope, you’re not missing anything. Currently there is no explicit hook to 
provide custom transformation logic on outbound or inbound data movement. We’re 
looking at ways to better support this. Can you tell me more about the types of 
processing you’d like to do? For example, is XQSync’s 1 document in-1 document 
out the right granularity? Any details would be much appreciated.

Justin



Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.commailto:justin.mak...@marklogic.com
www.marklogic.comhttp://www.marklogic.com/



On Apr 26, 2013, at 1:10 PM, Steiner, David J. (LNG-DAY) 
david.j.stei...@elsevier.commailto:david.j.stei...@elsevier.com wrote:

When I need to export a portion of a document, or manipulate it in some way, I 
generally use XQSync to and put some XQuery at INPUT_MODULE_URI, then run 
XQSync.

In looking at mlcp, I don’t really see an option to transform the data when 
exporting it to a file system.  Am I missing something?

Thanks,
David
___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Mlcp Can it load directly from mq ?

[Justin Makeig]

There is no public interface today in mlcp to load data from an InputStream, 
for example. We’re looking at how we can make the input and output interfaces 
more extensible in a future release. While not a great answer, in the meantime, 
mlcp is open source (Apache 2.0). I’ve seen several instances of users 
modifying the code slightly to affect a specific behavior. If you do fork it, 
I’d be very interested in understanding your changes to 1) improve the 
out-of-the-box functionality and 2) make sure we have the right extensibility 
hooks. 

As always, feedback is much appreciated.

Justin

On Apr 19, 2013, at 12:32 AM, MarkLogic User marklogicu...@yahoo.com
 wrote:

 
 Hi,
 
 We are considering using MLCP to load hundreds of millions of xml docs into 
 the MarkLogic database.
 Now we get data from an MQ. So in order to ingest this data into MarkLogic, 
 we first extract and store the files into file system and then use mlcp to 
 load into MarkLogic database.
 But storing in file system is an Extra Overhead which we want to avoid. So is 
 there any way that we can use mlcp to load data straightway from an MQ to the 
 MarkLogic database.
 
 Thanks..
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Record Loader - not able to load binary files (pdf, images)

[Justin Makeig]

Which version of MarkLogic are you using? Which version of RecordLoader? What’s 
the exact command you’re issuing to invoke the JAR? What’s the error message 
you’re getting? If you’re using MarkLogic 6, you might also take a look at mlcp 
http://docs.marklogic.com/guide/ingestion/content-pump.

Justin

Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.commailto:justin.mak...@marklogic.com
www.marklogic.comhttp://www.marklogic.com/



On Apr 19, 2013, at 11:11 AM, Mohanraj Chozhan 
mohanraj_choz...@infosys.commailto:mohanraj_choz...@infosys.com
 wrote:

Hi,

I am using the ML record Loader. But unable to load the binary files pdf, 
images into ML repository.

In the record loader properties file we added

DOCUMENT_TYPE=binary

But still facing the issue to load it.

Can someone help me out on this.

Regards,
Mohanraj

 CAUTION - Disclaimer *
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are 
not
to copy, disclose, or distribute this e-mail or its contents to any other 
person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has 
taken
every reasonable precaution to minimize this risk, but is not liable for any 
damage
you may sustain as a result of any virus in this e-mail. You should carry out 
your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this 
e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS End of Disclaimer INFOSYS***


___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] MarkLogic Content Pump not working with Million Files

[Justin Makeig]

Can you please provide the command line directive that you used (sans password, 
of course)?

Justin


Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.commailto:justin.mak...@marklogic.com
www.marklogic.comhttp://www.marklogic.com/



On Apr 10, 2013, at 4:14 AM, MarklogicUser 
marklogicu...@yahoo.commailto:marklogicu...@yahoo.com
 wrote:

Hi,

When I try to use mlcp to ingest files into MarkLogic, it works successfully 
for some files  say a few hundred.
But when I try to do that for a Unix directory having 1 million files. It 
simply hangs..doesn't load anything.
We are using MarkLogic6.0 and the latest mlcp as well.
Please suggest..Thanks..
___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Redirect from rewriter?

[Justin Makeig]

This isn’t possible from the rewriter. As you’ve suggested, you’ll have to 
create a main module that does the redirection and rewrite to that.

Justin

Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com



On Feb 26, 2013, at 2:50 PM, Will Thompson wthomp...@jonesmcclure.com
 wrote:

 We want to use the rewriter on one App Server to conditionally redirect to a 
 REST server if it matches a URL pattern. However, the redirect doesn't seem 
 to do anything. Do I have to create an intermediate module on the first App 
 Server that then redirects to the second, or is these a way to do it directly?
 
 -Will
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Getting Bad word size during restore db

[Justin Makeig]

Idus, thanks for the feedback. I’ve responded inline below.


On Feb 25, 2013, at 2:08 PM, Ildus Hakov iha...@iqsolutions.com wrote:

 Thanks for the responses. But looks like to move dbs and servers to prod or 
 qa from the dev env, not an easy task:
 - backups very sensitive, does not work on diff. platforms or something else. 
 (I have both platform winnt, architecture one Intel another And).

Backups are platform-specific. To move database contents across platforms you 
should look at mlcp’s copy command 
http://docs.marklogic.com/guide/ingestion/content-pump#id_42430. It can 
migrate documents and metadata (permissions, collections, quality) between two 
databases without having to maintain an intermediate export or write any custom 
code.


 - export/import supports only configurations.

Yes, the Configuration Manager in v6 only supports app server and database 
settings, such as indexes and fields. To bootstrap forests and hosts, for 
example, you’ll have to invoke the Admin API 
http://docs.marklogic.com/guide/admin-api/usingAPI#chapter.


 - the application to be moved needs to be packaged manually, there is no way 
 to create deployment file.

Yes, as of v6 there is no way to package up application code. Best practice for 
production environments, however, is to keep code in a modules database. You 
can migrate this content using mlcp copy just like you’d do with other data. 
Same with Security, Triggers, and Schemas.

 
 So I have to manually create all dbs, servers.
 So as for a data I had to write the xqy to save all docs from db to disk, 
 copy them to target box, then run document-insert.
 Then I believe I can load config files. Ahhh:(
 Then manually save all application files, copy and insert to the target or 
 may be I can do the same thing as I did for the data?
 
 And then the all these steps for production. 
 
 ildus

We’re actively working on making these steps simpler and more automated (and 
automatable, if that’s a word). Your feedback is greatly appreciated.

Justin Makeig

Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com


 
 -Original Message-
 From: general-boun...@developer.marklogic.com 
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of 
 general-requ...@developer.marklogic.com
 Sent: Monday, February 25, 2013 3:00 PM
 To: general@developer.marklogic.com
 Subject: General Digest, Vol 104, Issue 83
 
 Send General mailing list submissions to
   general@developer.marklogic.com
 
 To subscribe or unsubscribe via the World Wide Web, visit
   http://developer.marklogic.com/mailman/listinfo/general
 or, via email, send a message with subject or body 'help' to
   general-requ...@developer.marklogic.com
 
 You can reach the person managing the list at
   general-ow...@developer.marklogic.com
 
 When replying, please edit your Subject line so it is more specific than Re: 
 Contents of General digest...
 
 
 Today's Topics:
 
   1. Re: Single transaction and rollback on   failure (Michael Blakeley)
   2. Re: Getting Bad word size during restore db (Michael Blakeley)
   3. Re: Getting Bad word size during restore db (Danny Sokolsky)
   4. Re: Getting Bad word size during restore db (Michael Blakeley)
   5. Re: Asyncronous Status Updates (Tim)
 
 
 --
 
 Message: 1
 Date: Mon, 25 Feb 2013 09:53:54 -0800
 From: Michael Blakeley m...@blakeley.com
 Subject: Re: [MarkLogic Dev General] Single transaction and rollback
   on  failure
 To: MarkLogic Developer Discussion general@developer.marklogic.com
 Message-ID: 5f790153-593a-40e9-8efb-69e4e5450...@blakeley.com
 Content-Type: text/plain; charset=iso-8859-1
 
 Which version of the server are you using?
 
 You haven't said what's going wrong. What's the problem? If you have an error 
 message it's always a good idea to paste that in.
 
 I suspect that you're seeing something like XDMP-MULTIDBSTMT: Cannot process 
 different-database requests with same-statement isolation. If so the error 
 message says it all: you can't update two different databases in the same 
 statement. This is covered at 
 http://docs.marklogic.com/guide/app-dev/transactions#id_24388
 
 You may not use same-statement isolation when using the database option of 
 xdmp:eval orxdmp:invoke to specify a different database than the database in 
 the calling statement's context. If your eval/invoke code needs to use a 
 different database, use different-transaction isolation.
 
 Using different-transaction isolation means that you're really committing two 
 transactions, each atomic in its own database but discrete from each other. 
 You can try to structure your update sequence so that the first update will 
 be the sensitive one, and will automatically keep the second one from running 
 if it throws an error. But I can't think of a way to update two different 
 databases in a single ACID transaction. I

Re: [MarkLogic Dev General] Big Data

[Justin Makeig]

Pradeep,
I appreciate your inquiry. Can you be a little more specific, though? What type 
of problem are you trying to solve? What does “Big Data” mean here? How else 
have you thought about tackling this problem? Any additional context will help 
to provide more relevant guidance.
In the meantime, for a basic technical background on MarkLogic, I’d start with 
the videos and articles on the developer web site 
http://developer.marklogic.com/.

Justin




Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.commailto:justin.mak...@marklogic.com
www.marklogic.comhttp://www.marklogic.com/



On Feb 18, 2013, at 1:05 AM, Pradeep Maddireddy 
pradeep.marklo...@gmail.commailto:pradeep.marklo...@gmail.com
 wrote:

Hi..!

We would like to explore the Big Data capabilities of Marklogic. Please guide 
me in finding the right path. Are there any tutorial available?

Thanks
Pradeep Maddireddy
___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Help - Will Marklogic support calling query with parameters

[Justin Makeig]

Shibu, I’d start with the Java API that comes as part of MarkLogic 6. Among 
other things, it allows you to manage documents and perform searches. You can 
find the “Java Developer’s Guide” at http://docs.marklogic.com/guide/java. 
You can also download the JARs (or access the Maven repository) from 
http://developer.marklogic.com/products/java. The download page has links to 
other helpful getting started guides. 
If the built-in Java or REST APIs don’t do what you need, you can also build 
your own HTTP services, as it appears you’ve started below. However, for common 
tasks, such as search, you’ll be duplicating the functionality of REST and Java 
APIs that we’ve included in MarkLogic 6.

Justin


Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com



On Jan 23, 2013, at 12:31 AM, Shibu N n.sh...@yahoo.com
 wrote:

 
 Hi Friends,
 
 I'm relatively new to marklogic and I have a requirement where I'll have to 
 call a query on the server with parameters using the java API. I'm not sure 
 of the best approach to do this either. whether to use java API/ REST API's.
 
 The query is of the form http://localhost:8010/search.xq?text=searchtext
 
 The query search.xq simply performs a search based on searchtext and returns 
 the results,
 
 Any tips would be helpful,
 
 Thanks  Regards
 
 Shibu
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



smime.p7s
Description: S/MIME cryptographic signature
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Help - Will Marklogic support calling query with parameters

[Justin Makeig]

(: search.xq :)
cts:search(collection(), cts:word-query(xdmp:get-request-field(text)))

You’ll need to save the above as file in the modules root of an HTTP app server 
on port 8010 http://docs.marklogic.com/guide/admin/http. Then you can use 
your URL below to make a request from any HTTP client.
The above is a simplified illustration of search, a word query constructor, and 
HTTP request handling. You'd probably want to do some query parsing and 
pagination in a typical search app. (Another plug: The Search API, REST API, 
and Java API do all of this for you.)

Justin



Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.com
www.marklogic.com



On Jan 23, 2013, at 10:22 PM, Shibu N n.sh...@yahoo.com
 wrote:

 Hi,
 
 I think I have not been able to convey my requirement clearly. I have read 
 through the API's and have understood that marklogic allows you to search 
 documents based on properties. In my case I already have a query search.xq 
 installed on the server. I have to invoke 
 http://localhost:8010/search.xq?text=search text. My question is how do I 
 specify the search query search.xq with parameter key text and value search 
 text. 
 
 In simple terms what will be the java code for invoking 
 http://localhost:8010/search.xq?text=search text. The server is NOT enabled 
 for rest API's. Is that the reason?
 
 I'm sorry if you have given me directions and I haven't understood.
 
 Regards
 
 Shibu
 
 
 From: Erik Hennum erik.hen...@marklogic.com
 To: MarkLogic Developer Discussion general@developer.marklogic.com 
 Sent: Wednesday, January 23, 2013 9:58 PM
 Subject: Re: [MarkLogic Dev General] Help - Will Marklogic support calling 
 query with parameters
 
 Hi, Shibu:
 
 If the built-in search capabilities of the REST API aren't sufficient, you
 can extend the REST API write new resource services in XQuery.  Please
 see:
 
 http://docs.marklogic.com/guide/rest-dev/extensions
 
 
 Erik Hennum
 
 From: general-boun...@developer.marklogic.com 
 [general-boun...@developer.marklogic.com] on behalf of Shibu N 
 [n.sh...@yahoo.com]
 Sent: Wednesday, January 23, 2013 2:21 AM
 To: MarkLogic Developer Discussion
 Subject: Re: [MarkLogic Dev General] Help - Will Marklogic support calling 
 query with parameters
 
 Thanks Justin for that Info, 
 
 I did do some reading into the java API's. I have seen examples searching for 
 documents, but in this case I'm trying to invoke the search.xq file using the 
 API. Just wanted to know if this is possible. The search.xq file is like a 
 stored procedure which would give me the required results by accepting the 
 parameters for searchtext.
 
 Thanks  Regards
 
 Shibu
 
 --- On Wed, 1/23/13, Justin Makeig justin.mak...@marklogic.com wrote:
 
 From: Justin Makeig justin.mak...@marklogic.com
 Subject: Re: [MarkLogic Dev General] Help - Will Marklogic support calling 
 query with parameters
 To: MarkLogic Developer Discussion general@developer.marklogic.com
 Date: Wednesday, January 23, 2013, 12:48 AM
 
 Shibu, I’d start with the Java API that comes as part of MarkLogic 6. Among 
 other things, it allows you to manage documents and perform searches. You can 
 find the “Java Developer’s Guide” at http://docs.marklogic.com/guide/java. 
 You can also download the JARs (or access the Maven repository) from 
 http://developer.marklogic.com/products/java. The download page has links 
 to other helpful getting started guides. 
 If the built-in Java or REST APIs don’t do what you need, you can also build 
 your own HTTP services, as it appears you’ve started below. However, for 
 common tasks, such as search, you’ll be duplicating the functionality of REST 
 and Java APIs that we’ve included in MarkLogic 6.
 
 Justin
 
 
 Justin Makeig
 Director, Product Management
 MarkLogic Corporation
 justin.mak...@marklogic.com
 www.marklogic.com
 
 
 
 On Jan 23, 2013, at 12:31 AM, Shibu N n.sh...@yahoo.com
  wrote:
 
 
 Hi Friends,
 
 I'm relatively new to marklogic and I have a requirement where I'll have to 
 call a query on the server with parameters using the java API. I'm not sure 
 of the best approach to do this either. whether to use java API/ REST API's.
 
 The query is of the form http://localhost:8010/search.xq?text=searchtext
 
 The query search.xq simply performs a search based on searchtext and returns 
 the results,
 
 Any tips would be helpful,
 
 Thanks  Regards
 
 Shibu ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 
 -Inline Attachment Follows-
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Role based security in ml

[Justin Makeig]

I don’t know the details of how Corona assigns permissions to documents, but it 
sounds like you don’t have a read permission for the corona-dev role on any of 
your documents. Permissions associate capabilities (read, insert, update, and 
execute) with a role (e.g. corona-dev). Permissions are assigned to documents 
(usually at load time). Take a look at the “Document Permissions” section of 
the Security Guide in the docs 
http://docs.marklogic.com/guide/security/permissions#id_85471 to understand 
what’s happening under the covers.

Justin


Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.commailto:justin.mak...@marklogic.com
Phone: +1 650 655 2387
www.marklogic.comhttp://www.marklogic.com/



On Dec 13, 2012, at 9:39 PM, manoj viswanadha 
manoj.viswana...@gmail.commailto:manoj.viswana...@gmail.com wrote:

Hi Geert,

Thanks for your quick reply. I have created a user and gave corona-dev role and 
still i am not able
to get the results until i give him admin role. Am i missing any configuration 
settings from corona
side?

Thanks,
Manoj.

On Fri, Dec 14, 2012 at 1:28 AM, Geert Josten 
geert.jos...@dayon.nlmailto:geert.jos...@dayon.nl wrote:
Hi Manoj,

The Corona setup includes creation of two corona roles: corona-admin and 
corona-dev. The corona-dev role is probably what you are looking for. Just 
create MarkLogic users that have that role, and that should suffice.

See also: https://github.com/marklogic/Corona/wiki/Corona%20Overview

Kind regards,
Geert

Van: 
general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com
 
[mailto:general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com]
 Namens manoj.viswanadha
Verzonden: donderdag 13 december 2012 19:40
Aan: general@developer.marklogic.commailto:general@developer.marklogic.com
Onderwerp: [MarkLogic Dev General] Role based security in ml


Hi all,

I am using ml security and created few roles and users. My requirement is to 
restrict the http servers where corona is configured for specific users. I want 
specific users to access the corona search only and will have no other 
privilages.i have followed the steps mentioned in ml security pdf and created 
execute privilage and role and assiged that to user and in http admin 
configuration chaged the previlage and given security as digest. I have tried 
diffen
ent ways but unable to achieve above. Can anyone help me out to create a user 
who can have only acess to http sever and able to hit the corona search using 
restful webservices with out giving admin role.
Thanks,
Manoj.
Sent from Samsung Galaxy Note

___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Odd app server behaviour

[Justin Makeig]

Declan,
Can you help me understand how the default collation “broke in Query Console”? 
Was the Query Console UI not able to handle that? Were the results not as 
expected? Any details would help to improve Query Console. Thanks.

Justin


Justin Makeig
Director, Product Management
MarkLogic Corporation
justin.mak...@marklogic.commailto:justin.mak...@marklogic.com
Phone: +1 650 655 2387
www.marklogic.comhttp://www.marklogic.com/



On Nov 30, 2012, at 12:28 PM, Declan Newman 
declan.new...@semantico.commailto:declan.new...@semantico.com wrote:

Thanks to all for your replies.

That has fixed the problem.

A colleague managed to fix it by using the:

declare default collation http://marklogic.com/collation/codepoint;;

But that broke in the Query Console - so this is best outcome. That also 
explains why a range was working for an xs:date.

I'm still unsure about why this was working from the Roxy tests - but I can 
live with not knowing, as I can checkin the code now :-)

Thanks again,

Dec


On 30 Nov 2012, at 19:42, Danny Sokolsky 
danny.sokol...@marklogic.commailto:danny.sokol...@marklogic.com wrote:

As Mike said, the collation is a good thing to double-check.  It is best 
practice to specify the collation in the range constraint, that way you will 
not have to worry about default collation.  So make sure your constraint has 
the collation specified.  Something like:

options xmlns=http://marklogic.com/appservices/search;
constraint name=name
   range type=xs:string collation=http://marklogic.com/collation/;
 element ns= name=name/
   /range
/constraint
/options

Another hint is to look very closely at the entire error you are getting—it 
often has useful info that is easy to miss.

If you still cannot figure it out, post some code that we can look at and then 
maybe we can go from there.

-Danny

From: 
general-boun...@developer.marklogic.commailto:general-boun...@developer.marklogic.com
 
[mailto:general-boun...@developer.marklogic.commailto:boun...@developer.marklogic.com]
 On Behalf Of Mike Sokolov
Sent: Friday, November 30, 2012 9:34 AM
To: MarkLogic Developer Discussion
Cc: Fahad Khan
Subject: Re: [MarkLogic Dev General] Odd app server behaviour

You might also have a different default collation - that's my guess, since it 
is configurable per-appserver

On 11/30/2012 10:07 AM, Fahad Khan wrote:
Hello,

Whenever I have this issue it turns out to be name-space related. I am guessing 
you have double checked that too?

Fahad.



On 11/30/2012 02:35 PM, Declan Newman wrote:
Hello,

I'm having a very odd problem with ML v6.0-2.

I have my range element indexes all set-up for the elements that I wish to 
facet on.

I am able to call the function that calls

search:search($searchTerm, $search-options, ($page - 1) * $itemsPerPage + 1, 
$itemsPerPage)}

from qconsole and the facets are returned correctly - but as soon as I call 
this function from within an app I receive the error No element range index … 
with the list of indexes that I have double checked exist as 
element-range-index.

I have also double/triple/quadruple-checked that the app is talking to the 
correct database.

Is there something I'm missing here? I'm currently stumped.

Thanks,

Dec



Declan Newman, Development Team Lead,
Semantico, Floor 1, 21-23 Dyke Road, Brighton BN1 3FE
mailto:declan.new...@semantico.com
tel:+44-1273-358247





___

General mailing list

General@developer.marklogic.commailto:General@developer.marklogic.com

http://developer.marklogic.com/mailman/listinfo/general









___

General mailing list

General@developer.marklogic.commailto:General@developer.marklogic.com

http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general


Declan Newman, Development Team Lead,
Semantico, Floor 1, 21-23 Dyke Road, Brighton BN1 3FE
mailto:declan.new...@semantico.com
tel:+44-1273-358247

___
General mailing list
General@developer.marklogic.commailto:General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general