Re: [MarkLogic Dev General] DataHub app digest authentication with MarkLogic 9

[Ryan Dew]

Hi Ron,

You can find the changes that fixed this issue at
https://github.com/marklogic-community/slush-marklogic-node/pull/520/files

Basically, when performing the path as part of the digest authentication
was missing the "/v1" prefix.



On Thu, Apr 26, 2018 at 8:20 AM Ron Hitchens  wrote:

>
>I’ve inherited an application that was created by MarkLogic
> professional services.  It uses the the DataHub Framework and a Node.js app
> that was generated with Slush.  It was created on MarkLogic 8 and is
> working there.  I need to make several changes/improvements, one of which
> is to upgrade MarkLogic to v9.
>
>I’ve upgraded MarkLogic to v9 ok and the data load and harmonization
> stuff seems to be working ok now.  But I’m having no end of trouble getting
> the Node front-end app to talk to ML9.
>
>The problem seems to be the digest authentication protocol.  The app is
> able to complete login where it hits MarkLogic to validate the
> user/password and then stores it locally in a session for later search
> requests.  Those always fail with a 401 response.  I’ve done some tracing
> and it appears that it is indeed finding the session credentials and is
> setting the digest auth header as expected on the request to ML.
>
>Comparing the Authentication: header with the one sent by a successful
> curl request, the only difference of note seems to be that the “cnonce”
> feld is hex encoded by Node and is base64 encoded by curl.  I don’t know
> the digest protocol well enough to know if this is the core or the
> problem.  But in any case trying to track down how the authentication is
> generated is like being lost in a maze of twisty little passages, all
> Javascript.
>
>I found a small ML9 release note that digest authentication had been
> tightened up in ML9, which is probably the nub of the issue.  So my
> question is, how do I fix this?  How do I get the HTTP digest handshake
> working properly between this Node.js app and MarkLogic 9?
>
>Any help appreciated.
>
> 
> Ron Hitchens r...@overstory.co.uk, +44 7879 358212
>
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] MarkLogic Research

[Ryan Dew]

Hi Kara,

I previously responded to an Aaron from G2 Crowd. I'm employed by
MarkLogic. I know that you're looking for unbiased reviews. While I'd like
to be believe I'm unbiased, I'm not sure if my association disqualifies me
from reviewing MarkLogic through your system.

Thanks,
Ryan Dew

On Wed, Mar 23, 2016 at 9:30 AM, Kara Kennedy <k...@g2crowd.info> wrote:

> MarkLogic Listserv,
>
> I work for G2 Crowd, a user review site for software. I'm currently
> working on document databases research, and am having trouble finding users
> of MarkLogic. I want to include the product in my research report, and was
> hoping you could help with your expertise.
>
> If you can leave a review of MarkLogic, we can send a $15 Amazon gift card
> as a thank you.
>
> Link for review:
> https://www.g2crowd.com/contributor/opb-marklogic?rewards=true
>
> Reviews are reviewed and gift cards are sent within 3-5 days. If you know
> other users of the product, feel free to pass that offer on to them as well!
>
> Thanks for the help!
>
> -Kara Kennedy
>
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
>
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Socket Recieve exception

[Ryan Dew]

Browsers are often smart enough to encode spaces for you. What if you try
the following? Hopefully it gives you what you are looking for.

let $geturl := "
http://api.academicbenchmarks.com/rest/v3/standardspartner.id=devconnect04auth.signature=Tp1akyZwnn4hNcS91LQH1Rq7os9L4vGtxNHIz3GWT0s=auth.expires=1454652779865list=state,document,subject_docstate=insubject=math;
document.title=
<http://api.academicbenchmarks.com/rest/v3/standardspartner.id=devconnect04auth.signature=Tp1akyZwnn4hNcS91LQH1Rq7os9L4vGtxNHIz3GWT0s=auth.expires=1454652779865list=state,document,subject_docstate=insubject=mathdocument.title=Academic%20Standards>"
|| xdmp:url-encode("Academic Standards")



let $result :=

xdmp:http-get($geturl,



   60

 

      )

  return  $result

-Ryan Dew

On Mon, Feb 8, 2016 at 10:16 PM, <thangella.vinodre...@cognizant.com> wrote:

> Hi Team,
>
>
>
> Can anyone help on this. I am trying to send a get request using
> xdmp:http-get() and receiving below socket receive exception . I increased
> time out in the options as well.
>
>
> [1.0-ml] SVC-SOCRECV: xdmp:http-get("
> http://api.academicbenchmarks.com/rest/v3/standardspartner;,
> 60) -- Socket
> receive error: wait: Timeout (decodeResponseLine1)
>
>
>
>
>
> My code is
>
>
>
> xquery version "1.0-ml";
>
>
>
> let $geturl := "
> http://api.academicbenchmarks.com/rest/v3/standardspartner.id=devconnect04auth.signature=Tp1akyZwnn4hNcS91LQH1Rq7os9L4vGtxNHIz3GWT0s=auth.expires=1454652779865list=state,document,subject_docstate=insubject=math;
> document.title=Academic Standards
> <http://api.academicbenchmarks.com/rest/v3/standardspartner.id=devconnect04auth.signature=Tp1akyZwnn4hNcS91LQH1Rq7os9L4vGtxNHIz3GWT0s=auth.expires=1454652779865list=state,document,subject_docstate=insubject=mathdocument.title=Academic%20Standards>
> "
>
>
>
> let $result :=
>
> xdmp:http-get($geturl,
>
> 
>
>60
>
>  
>
>   )
>
>   return  $result
>
>
>
> When I am using ML7 I am getting the below exception
>
>
>
> 
>
> 406
>
> Not Acceptable
>
> 
>
> no-cache
>
> no-cache
>
> text/html; charset=utf-8
>
> close
>
> 652
>
> 
>
> 
>
> Text document
>
> 
>
>
>
> Request Error
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> Request Error (unsupported_protocol)
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> Your request used a protocol that is not currently supported.
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
>
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> For assistance, contact your network support team.
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> When I hit the same URL Using REST client I am getting the response and
> also in the above query if I remove *document.title* parameter I am
> getting the response from the above MarkLogic code. So I would like to know
> is there any specific parameters like document will not work with
> xdmp:http-get() function.
>
>
>
> Can anyone please help me on this ASAP.
>
>
>
>
>
> Thanks,
>
> Vinod.
>
>
>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>
> ___
> General mailing list
> General@developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
>
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Urgent request for assistance (please)

[Ryan Dew]

You can also try a helper library like this one:
https://github.com/ryanjdew/XQuery-XML-Memory-Operations
On Aug 1, 2014 6:17 PM, Pete Aven pete.a...@marklogic.com wrote:

 And to transform the doc using XQuery, take a look at the typeswitch
expression: http://docs.marklogic.com/guide/app-dev/typeswitch#id_65827



-pete



*From:* general-boun...@developer.marklogic.com [mailto:
general-boun...@developer.marklogic.com] *On Behalf Of *Joe Bryan
*Sent:* Friday, August 01, 2014 8:15 PM
*To:* MarkLogic Developer Discussion
*Subject:* Re: [MarkLogic Dev General] Urgent request for assistance
(please)



Hi Paul,



The xdmp:node-* functions only work on documents stored in the database. To
update a file on disk, you'll need to reconstruct the entire document as
you want, and then save it using xdmp:save.



Can I ask why you've taken this approach? I think you'll be much better
served by querying, transforming, and updating documents that are stored in
the database. There's a much larger API surface available to you, and
you'll be able to leverage the universal index.



Thanks.



-jb



*From: *Paul Farrell pauldfarr...@hotmail.com
*Reply-To: *MarkLogic Developer Discussion general@developer.marklogic.com
*Date: *Friday, August 1, 2014 at 7:55 PM
*To: *general@developer.marklogic.com general@developer.marklogic.com
*Subject: *[MarkLogic Dev General] Urgent request for assistance (please)



Hi,



I am desperately hoping someone out there may be able to help me out with
an xquery app that I am building.



I have only just started with Marklogic and XQuery. It's going fairly well,
however I am having a really tough time in modifying the content of one of
my XML documents. I just cannot seem to get a change to an element to pick
up. Here's my process (I have had to take things back as basic as I could
just to try and get it working):



1. In Query console I have one tab open which queries for the contents of
one XML doc



xquery version 1.0-ml;

declare namespace html = http://www.w3.org/1999/xhtml;;

xdmp:document-get(C:/Users/Paul/Documents/MarkLogic/xml/ppl/ppl/jdbc_ppl_3790.xml)



2. This brings back the document as below



?xml version=1.0 encoding=UTF-8?

document

meta

rm_mimetype

/rm_mimetype

rm_hasattachments


false

/rm_hasattachments

rm_attachmentcount

...

3790

/ppl_id

ppl_name


Victoria Wilson

/ppl_name



   3. I now want to update the ppl_name element using XQuery but it's
just not happening. Here's the XQuery:

xquery version 1.0-ml;

declare namespace html = http://www.w3.org/1999/xhtml;;





let $docxml :=

xdmp:document-get(C:/Users/Paul/Documents/MarkLogic/xml/ppl/ppl/jdbc_ppl_3065.xml)/document/meta/ppl_name

return

  for $node in $docxml/*

  let $target :=
xdmp:document-get(C:/Users/Paul/Documents/MarkLogic/xml/ppl/ppl/jdbc_ppl_3790.xml)/document/meta/*[fn:name()
= fn:name($node)]

  return

  xdmp:node-replace($target, $node)



--- I am basically looking to replace the ppl_name element in the target
(3790) with the ppl_name element from the source (3065).



4. I run the Xquery - it completes without error (making me thing it has
worked) - return value reads your query returned an empty sequence



5. I then go back to the same tab as I used in step 1 and re-run the XQuery
used in step 1. The doc (3790) comes back but it STILL has Victoria Wilson
as the ppl_name





Can anyone please help? Perhaps the change needs committing? I just don't
know.



Thanks for reading

Paul



___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] XSLT check if result document has been created

[Ryan Dew]

I believe you should be able to do something like the following to get what
you want:

xsl:template match=//db:informalfigure[descendant::db:imagedata and
@role='figure']



xsl:variable name=curImage select=
substring-after(.//@fileref,'/')/

xsl:variable name=id select=
$imageMetaData//image[name=string-join(($curISBN,$curImage),'/')]/id/
xsl:if test=not(doc-available(
xdmp:resolve-uri(string-join(('out/',$id,'.xml'),''),xdmp:node-uri(.
   !--This check fails, am I doing it incorrectly or is it the way xslt
processes document that makes it hard to check if the id have been
encountered before?--

xsl:result-document method=xml href=out/{$id}.xml indent=
yes


On Mon, Jul 28, 2014 at 9:08 AM, Mary Holstege mary.holst...@marklogic.com
wrote:


 I think you may be running afoul of URI resolution.
 Since the URI you are giving to doc-available is a relative
 URI, it will be resolved relative to the static base URI, which
 per XSLT is the URI of the stylesheet itself.

 //Mary

 On Mon, 28 Jul 2014 07:58:04 -0700, Erik Zander
 erik.zan...@studentlitteratur.se wrote:

  Hi All
 
  I’m working on an xslt transform where I’m extracting data about images
  from a document and put
  that data into result document, It all works fine except for when the
  same image occurs more than once as I then get conflicting uris,
 
  my code looks like this
  xsl:template match=//db:informalfigure[descendant::db:imagedata and
  @role='figure']
 
 
 
  xsl:variable name=curImage
  select=substring-after(.//@fileref,'/')/
 
  xsl:variable name=id
 
 select=$imageMetaData//image[name=string-join(($curISBN,$curImage),'/')]/id/
  xsl:if
  test=not(doc-available(string-join(('out/',$id,'.xml'),'')))
  !--This check fails, am I doing it incorrectly or is it the way xslt
  processes document that makes it hard to check if the id have been
  encountered before?--
 
  xsl:result-document method=xml href=out/{$id}.xml
  indent=yes
 
  Ideally I would like to be able to check if the result document have
  been created or not and after that decide if I want to update it with
  more information or just leave it be.
 
  Would appreciate any help on the subject
 
  Best regards
  Erik


 --
 Using Opera's revolutionary email client: http://www.opera.com/mail/
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] XDMP-NOTXN

[Ryan Dew]

Something could be ending your HTTP session, while your browser is holding
onto a cookie with the expired transaction information.

https://docs.marklogic.com/guide/app-dev/transactions#id_37297

Is there a use case that you have for carrying a transaction over multiple
calls? The transaction mode of query changes you over to multi-statement
transaction which need an explicit xdmp:commit() to end a transaction
despite in only being a query. Unless you have a good reason not to, you
could just add xdmp:commit() to the end of your code.

https://docs.marklogic.com/guide/app-dev/transactions#id_68542

As you make changes you may need to clear your cookies to truly know if
your changes fixed the problem.

-Ryan


On Fri, May 9, 2014 at 10:58 AM, William Sawyer wilby.saw...@gmail.comwrote:

 When I set update to false and transaction-mode to query on page it causes
 the page to error on everything other request.   Has anyone else seen this
 and know how to fix it?

 Thanks,
 -Will

 declare option xdmp:update false;
 declare option xdmp:transaction-mode query;

 500 Internal Server ErrorXDMP-NOTXN: No transaction with identifier
 295241064692587278[1.0-ml]

 ML: 7.0-2.3
 OS: Windows 7
 Single Host

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] xdmp:format-number broken?

[Ryan Dew]

That is odd, but it seems you shouldn't have a problem accomplishing what
you want just using a picture string like so: #,##0

let $value := 12312312
return (
  fn:format-number($value,#,##0)
)
RESULT
12,312,312


On Fri, Jan 17, 2014 at 4:38 PM, William Sawyer wilby.saw...@gmail.comwrote:

 Has anyone been able to to get xdmp:format-number grouping-separator and
 group-size parameters to work?

 let $value := 12312312
 let $picture := 0
 let $language := en
 let $letter-value := ()
 let $ordchar := ()
 let $zero-padding := ()
 let $grouping-separator := ,
 let $grouping-size := 3
 return (
   xdmp:format-number($value, $picture, $language, $letter-value, $ordchar,
 $zero-padding, $grouping-separator, $grouping-size)
 )
  RESULT 
 12312312


 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

[MarkLogic Dev General] Missing Incompatibility in ML 7 Docs

[Ryan Dew]

FYI, it seems that the results of fn:analyze-string are now namespaced
according to the standard even in '1.0-ml' mode of ML7. So the namespace is
http://www.w3.org/2005/xpath-functions; instead of 
http://www.w3.org/2009/xpath-functions/analyze-string; as it used to be. I
wouldn't consider it a bug, but it doesn't appear to be listed as an
incompatibility in the documentation located here:
http://docs.marklogic.com/guide/relnotes/chap4#chapter.

Also, sorry if this has already been brought up and I just missed it.

Thanks,
Ryan Dew
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Issue with xdmp:unquote if passing amp; Unexpected end of file at line 1

[Ryan Dew]

So it seems like it is an issue in the transfer to the page. Have you tried
placing the content in the request body. You can even set the type as
'application/xml' and you won't even have to use xdmp:unquote.


(: submission :)
let $options :=
options xmlns=xdmp:http
headers
content-typeapplication/xml/content-type
/headers
data{xdmp:quote($document)}/data
/options
return xdmp:http-post($searchUri, $options)

(: reception :)
declare variable $newdivisions := xdmp:get-request-body()/fwp:divisions;

$newdivisions

-Ryan Dew


On Wed, Nov 20, 2013 at 4:20 PM, Singh, Gurbeer 
gurbeer.si...@morganstanley.com wrote:

   xdmp:unquote



 I am passing  below string in Query string  and in XQY file we are trying
 to read like this



 *STRING *

 fwp:divisions xmlns:fwp='http://morganstanley.com/firmwidePolicy'

 fwp:bucket xmlns:fwp='http://morganstanley.com/firmwidePolicy'
 name='Technology Data'

 fwp:buArea xmlns:fwp='http://morganstanley.com/firmwidePolicy' level='0'
 tag='true' ![CDATA[Technology  Data]]/fwp:buArea

 fwp:bu xmlns:fwp='http://morganstanley.com/firmwidePolicy' level='1'
 tag='false' ![CDATA[Quality Assurance *amp;* Prod Mgmt]]/fwp:bu

 fwp:bu xmlns:fwp='http://morganstanley.com/firmwidePolicy' level='2'
 tag='true' ![CDATA[CPT CRGT QA]]/fwp:bu

 /fwp:bucket/fwp:divisions



 let $newdivisions := xdmp:get-request-field(division, )



 running on below query I am getting err


 xdmp:node-replace($doc//fwp:divisions,xdmp:unquote($newdivisions)/fwp:divisions)



 XDMP-DOCUNEOF: xdmp:unquote(fwp:divisions xmlns:fwp='
 http://morganstanley.com/firmwidePolic...;) -- Unexpected end of file at
 line 1 (cdata starting at line 1)





 If I am running above in CQ its working fine,



 declare namespace fwp=http://morganstanley.com/firmwidePolicy;;

 let $newdivisions :=fwp:divisions xmlns:fwp='
 http://morganstanley.com/firmwidePolicy'

 fwp:bucket xmlns:fwp='http://morganstanley.com/firmwidePolicy'
 name='Technology Data'

 fwp:buArea xmlns:fwp='http://morganstanley.com/firmwidePolicy' level='0'
 tag='true' ![CDATA[Technology  Data]]/fwp:buArea

 fwp:bu xmlns:fwp='http://morganstanley.com/firmwidePolicy' level='1'
 tag='false' ![CDATA[Quality Assurance amp; Prod Mgmt]]/fwp:bu

 fwp:bu xmlns:fwp='http://morganstanley.com/firmwidePolicy' level='2'
 tag='true' ![CDATA[CPT CRGT QA]]/fwp:bu

 /fwp:bucket/fwp:divisions

 

 return

 xdmp:unquote($newdivisions)







 My Service string is




 http://vi5tas01.ms.com:40001/services/ML_Migration.xqy?id=1431180contentType=contentType![CDATA[Procedure]]/contentTypedocumentType=fwp:documentType
 xmlns:fwp='http://morganstanley.com/firmwidePolicy' level='0' tag='true'
 ![CDATA[Procedures]]/fwp:documentTypedocumentType2=fwp:documentType2
 xmlns:fwp='http://morganstanley.com/firmwidePolicy' level='0' tag='true'
 ![CDATA[Procedure]]/fwp:documentType2division=fwp:divisions
 xmlns:fwp='http://morganstanley.com/firmwidePolicy'fwp:bucket
 xmlns:fwp='http://morganstanley.com/firmwidePolicy' name='Technology
 Data'fwp:buArea xmlns:fwp='http://morganstanley.com/firmwidePolicy'
 level='0' tag='true' ![CDATA[Technology  Data]]/fwp:buAreafwp:bu
 xmlns:fwp='http://morganstanley.com/firmwidePolicy' level='1' tag='false'
 ![CDATA[Quality Assurance amp; Prod Mgmt]]/fwp:bufwp:bu xmlns:fwp='
 http://morganstanley.com/firmwidePolicy' level='2' tag='true'
 ![CDATA[CPT CRGT
 QA]]/fwp:bu/fwp:bucket/fwp:divisionsnewElementKB=fwp:groupName
 xmlns:fwp='http://morganstanley.com/firmwidePolicy'Technology
 Risk/fwp:groupNamenewElementWF=fwp:workflowName xmlns:fwp='
 http://morganstanley.com/firmwidePolicy' /fwp:workflowName






 --

 NOTICE: Morgan Stanley is not acting as a municipal advisor and the
 opinions or views contained herein are not intended to be, and do not
 constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
 Street Reform and Consumer Protection Act. If you have received this
 communication in error, please destroy all electronic and paper copies and
 notify the sender immediately. Mistransmission is not intended to waive
 confidentiality or privilege. Morgan Stanley reserves the right, to the
 extent permitted under applicable law, to monitor electronic
 communications. This message is subject to terms available at the following
 link: http://www.morganstanley.com/disclaimers If you cannot access these
 links, please notify us by reply message and we will send the contents to
 you. By messaging with Morgan Stanley you consent to the foregoing.


 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] xdmp:filesystem-file-exists -- Any other option

[Ryan Dew]

You might want to checkout the following

Content DB:
http://docs.marklogic.com/fn:doc-available

Modules DB:
http://docs.marklogic.com/xdmp:uri-is-file

-Ryan Dew


On Tue, Aug 20, 2013 at 10:04 AM, Singh, Gurbeer 
gurbeer.si...@morganstanley.com wrote:

   I need check, my file exists in ML data base or not , I don’t want to
 load the fn:doc() and check.

 ** **

 xdmp:filesystem-file-exists works only filesystem. It’s not working for my
 webdav folder created for my ML database.

 ** **

 ** **

 let
 $path:=/ms/user/g/gurbeers/FPOL_ML_XQY/Policy/MLPortal/services/saveDocAndMetaN.xqy
 

 return

 xdmp:filesystem-file-exists($oldpath) (:true:)

 ** **

 ** **

 let $path:=/policies/otherDocs/1702386.xml

 return

 xdmp:filesystem-file-exists($path) (:false:)

 ** **

 ** **

 ~Gurbeer


 --

 NOTICE: Morgan Stanley is not acting as a municipal advisor and the
 opinions or views contained herein are not intended to be, and do not
 constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
 Street Reform and Consumer Protection Act. If you have received this
 communication in error, please destroy all electronic and paper copies and
 notify the sender immediately. Mistransmission is not intended to waive
 confidentiality or privilege. Morgan Stanley reserves the right, to the
 extent permitted under applicable law, to monitor electronic
 communications. This message is subject to terms available at the following
 link: http://www.morganstanley.com/disclaimers If you cannot access these
 links, please notify us by reply message and we will send the contents to
 you. By messaging with Morgan Stanley you consent to the foregoing.


 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] xdmp:filesystem-file-exists -- Any other option

[Ryan Dew]

Sorry I was mistaken on the second one. xdmp:uri-is-file:  Returns true if
a given URI refers to a file which exists on the current application
server. *Only returns true if the modules on the App Server is set to file
system;* always returns false if you are using a database for your modules

-Ryan


On Tue, Aug 20, 2013 at 10:25 AM, Ryan Dew ryan.j@gmail.com wrote:

 You might want to checkout the following

 Content DB:
 http://docs.marklogic.com/fn:doc-available

 Modules DB:
 http://docs.marklogic.com/xdmp:uri-is-file

 -Ryan Dew


 On Tue, Aug 20, 2013 at 10:04 AM, Singh, Gurbeer 
 gurbeer.si...@morganstanley.com wrote:

   I need check, my file exists in ML data base or not , I don’t want to
 load the fn:doc() and check.

 ** **

 xdmp:filesystem-file-exists works only filesystem. It’s not working for
 my webdav folder created for my ML database.

 ** **

 ** **

 let
 $path:=/ms/user/g/gurbeers/FPOL_ML_XQY/Policy/MLPortal/services/saveDocAndMetaN.xqy
 

 return

 xdmp:filesystem-file-exists($oldpath) (:true:)

 ** **

 ** **

 let $path:=/policies/otherDocs/1702386.xml

 return

 xdmp:filesystem-file-exists($path) (:false:)

 ** **

 ** **

 ~Gurbeer


 --

 NOTICE: Morgan Stanley is not acting as a municipal advisor and the
 opinions or views contained herein are not intended to be, and do not
 constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
 Street Reform and Consumer Protection Act. If you have received this
 communication in error, please destroy all electronic and paper copies and
 notify the sender immediately. Mistransmission is not intended to waive
 confidentiality or privilege. Morgan Stanley reserves the right, to the
 extent permitted under applicable law, to monitor electronic
 communications. This message is subject to terms available at the following
 link: http://www.morganstanley.com/disclaimers If you cannot access
 these links, please notify us by reply message and we will send the
 contents to you. By messaging with Morgan Stanley you consent to the
 foregoing.


 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] search:search and ordering by attribute on root element

[Ryan Dew]

Sorry, I used bad terminology. I said root element when I should have said
the element returned by cts:search. In our example, the searchable
expression that we set is //div so I believe that is what is being used
to construct a cts:search that looks like cts:search(//div, ...) so the
cts:search is actually returning the div element that matches the search.
That div happens to also be the element that has the order attribute on it
and I believe the order by is constructed so that it does order by
$result//div/@order when we really want order by
$result/descendant-or-self::div/@order. Since the div elements we are
looking for aren't root elements we don't have the option to jump up to the
document level with something like fn:collection().

I'm thinking, for now, our best option would probably be to use
search:parse to build the cts:queries to pass into cts:search and do our
own ordering logic.

Thanks,
Ryan Dew


On Thu, May 2, 2013 at 10:20 AM, Colleen Whitney 
colleen.whit...@marklogic.com wrote:

   The other issue that commonly causes sorting to look confusing is when
 you have multiple instances of an element in the same document.  Is there
 any possibility that this is the case with your data?
   --
 *From:* general-boun...@developer.marklogic.com [
 general-boun...@developer.marklogic.com] on behalf of Erik Hennum [
 erik.hen...@marklogic.com]
 *Sent:* Thursday, May 02, 2013 6:49 AM
 *To:* MarkLogic Developer Discussion
 *Subject:* Re: [MarkLogic Dev General] search:search and ordering by
 attribute on root element

   Hi, Ryan:

 cts:search() returns documents, so // will match the root element.

 In my simple test, search:search() did sort on an attribute on the root
 element.

 A few simple things to check -- is the div element in the XHTML
 namespace?  Is there a range index on the attribute in the same collation
 as the query?


  Erik Hennum

   --
 *From:* general-boun...@developer.marklogic.com [
 general-boun...@developer.marklogic.com] on behalf of Ryan Dew [
 ryan.j@gmail.com]
 *Sent:* Wednesday, May 01, 2013 3:02 PM
 *To:* MarkLogic Developer Discussion
 *Subject:* [MarkLogic Dev General] search:search and ordering by
 attribute on root element

   Hello everyone,

 We have scenario where we are trying to sort on an attribute on the root
 element being returned. The simple use case is shown below. The div
 elements that we are returning have an order attribute.

 let $options :=
 options xmlns=http://marklogic.com/appservices/search;
 searchable-expression//div/searchable-expression
 transform-results apply=raw/
 sort-order type=xs:string direction=ascending
 element ns= name=div/
 attribute ns= name=order/
 annotationchronology/annotation
 /sort-order
   /options

  return search:search(faith,$options,1,10)

 We don't get the results we would expect. After looking at the
 search:search code it appears that the root element isn't included in the
 XPath expression generated for the order by. It generates
 $result//div/@order, but I would expect something like:
 $result/descendant-or-self::div/@order. (Sorry if I'm digging more than I
 should.) Has anyone else encountered this problem and worked around it?

 Thanks,
 Ryan Dew

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] search:search and ordering by attribute on root element

[Ryan Dew]

I'll look into that. Thanks everyone!


On Thu, May 2, 2013 at 11:59 AM, Erik Hennum erik.hen...@marklogic.comwrote:

  Hi, Ryan:

 A couple of belated thoughts besides revising the XPath for searchable
 expression.

 If you're using ML6 and you can express the query as a structured search
 instead of a string search, you can move the existential query for div out
 of the searchable expression:

 search:resolve(
 query xmlns=http://marklogic.com/appservices/search;
   element-constraint-query
   constraint-namedivcontainer/constraint-name
   and-query/
   /element-constraint-query
 /query,
 options xmlns=http://marklogic.com/appservices/search;
   constraint name=divcontainer
   element-query ns= name=root/
   /constraint
 ...
 /options)

 For string search, you can also express the existential query on div with
 an equivalent cts:element-query() in the search:additional-query element.


 Hoping that's useful,


  Erik Hennum

   --
 *From:* general-boun...@developer.marklogic.com [
 general-boun...@developer.marklogic.com] on behalf of Erik Hennum [
 erik.hen...@marklogic.com]
 *Sent:* Thursday, May 02, 2013 9:50 AM

 *To:* MarkLogic Developer Discussion
 *Subject:* Re: [MarkLogic Dev General] search:search and ordering by
 attribute on root element

   Hi, Ryan:

 Any chance your searchable expression could qualify the parent of div
 (with an XPath that includes a predicate that confirms the existence of a
 div child)?


  Erik Hennum
   --
 *From:* general-boun...@developer.marklogic.com [
 general-boun...@developer.marklogic.com] on behalf of Ryan Dew [
 ryan.j@gmail.com]
 *Sent:* Thursday, May 02, 2013 9:38 AM
 *To:* MarkLogic Developer Discussion
 *Subject:* Re: [MarkLogic Dev General] search:search and ordering by
 attribute on root element

   Sorry, I used bad terminology. I said root element when I should have
 said the element returned by cts:search. In our example, the searchable
 expression that we set is //div so I believe that is what is being used
 to construct a cts:search that looks like cts:search(//div, ...) so the
 cts:search is actually returning the div element that matches the search.
 That div happens to also be the element that has the order attribute on it
 and I believe the order by is constructed so that it does order by
 $result//div/@order when we really want order by
 $result/descendant-or-self::div/@order. Since the div elements we are
 looking for aren't root elements we don't have the option to jump up to the
 document level with something like fn:collection().

  I'm thinking, for now, our best option would probably be to use
 search:parse to build the cts:queries to pass into cts:search and do our
 own ordering logic.

  Thanks,
 Ryan Dew


 On Thu, May 2, 2013 at 10:20 AM, Colleen Whitney 
 colleen.whit...@marklogic.com wrote:

   The other issue that commonly causes sorting to look confusing is when
 you have multiple instances of an element in the same document.  Is there
 any possibility that this is the case with your data?
   --
 *From:* general-boun...@developer.marklogic.com [
 general-boun...@developer.marklogic.com] on behalf of Erik Hennum [
 erik.hen...@marklogic.com]
 *Sent:* Thursday, May 02, 2013 6:49 AM
 *To:* MarkLogic Developer Discussion
 *Subject:* Re: [MarkLogic Dev General] search:search and ordering by
 attribute on root element

Hi, Ryan:

 cts:search() returns documents, so // will match the root element.

 In my simple test, search:search() did sort on an attribute on the root
 element.

 A few simple things to check -- is the div element in the XHTML
 namespace?  Is there a range index on the attribute in the same collation
 as the query?


  Erik Hennum

   --
 *From:* general-boun...@developer.marklogic.com [
 general-boun...@developer.marklogic.com] on behalf of Ryan Dew [
 ryan.j@gmail.com]
 *Sent:* Wednesday, May 01, 2013 3:02 PM
 *To:* MarkLogic Developer Discussion
 *Subject:* [MarkLogic Dev General] search:search and ordering by
 attribute on root element

   Hello everyone,

 We have scenario where we are trying to sort on an attribute on the root
 element being returned. The simple use case is shown below. The div
 elements that we are returning have an order attribute.

 let $options :=
 options xmlns=http://marklogic.com/appservices/search;
 searchable-expression//div/searchable-expression
 transform-results apply=raw/
 sort-order type=xs:string direction=ascending
 element ns= name=div/
 attribute ns= name=order/
 annotationchronology/annotation
 /sort-order
   /options

  return search:search(faith,$options,1,10)

 We don't get the results we would expect. After looking at the
 search:search code it appears that the root element isn't included in the
 XPath expression

Re: [MarkLogic Dev General] How to test whether doc exists

[Ryan Dew]

Interesting. The results could be affected by having the document already
in the caches. You might want to include non-existing documents in your
test. You could also try:

xdmp:exists(cts:search(fn:collection(), cts:document-query($URI)))
On Apr 24, 2013 10:34 PM, Tim Finney t...@tfinney.net wrote:

  Hi Ryan,

 Here are the profile times I get using qconsole to test for existence of a
 (known to exist) doc at $URI:

 exists(doc($URI)): 80 us (microseconds)
 doc-available($URI): 200 us
 cts:uri-match($URI): 42000 us

 Best,

 Tim Finney

 On 04/25/2013 12:22 PM, Tim wrote:

  What about searching the uri lexicon?

 ** **

 *From:* general-boun...@developer.marklogic.com [
 mailto:general-boun...@developer.marklogic.comgeneral-boun...@developer.marklogic.com]
 *On Behalf Of *Ryan Dew
 *Sent:* Thursday, April 25, 2013 12:04 AM
 *To:* MarkLogic Developer Discussion
 *Subject:* Re: [MarkLogic Dev General] How to test whether doc exists

 ** **

 fn:doc-available is pretty fast.

 On Apr 24, 2013 9:36 PM, Tim Finney t...@tfinney.net wrote:

 Hi All,

 What is the fastest way to test for existence of a document in the
 database?

 This is the fastest way I know:

 let $URI := /path/to/doc.xml
 return exists(doc($URI))

 It takes about 75 us (microseconds) to return an answer according to the
 profile tab of qconsole.

 Best,

 Tim Finney
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


 ___
 General mailing 
 listGeneral@developer.marklogic.comhttp://developer.marklogic.com/mailman/listinfo/general



 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] How to test whether doc exists

[Ryan Dew]

fn:doc-available is pretty fast.
On Apr 24, 2013 9:36 PM, Tim Finney t...@tfinney.net wrote:

 Hi All,

 What is the fastest way to test for existence of a document in the
 database?

 This is the fastest way I know:

 let $URI := /path/to/doc.xml
 return exists(doc($URI))

 It takes about 75 us (microseconds) to return an answer according to the
 profile tab of qconsole.

 Best,

 Tim Finney
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] How to program the order by clause in FLWOR?

[Ryan Dew]

I was thinking you would have to do something like this (obviously very
tailored to the example given):

let $sort-options := fn:tokenize(a descending,  )
let $sort-node := $sort-options[1]
let $sort-direction := $sort-options[2]
for $row in
(rowa0/ab5/b/row,rowa3/ab4/b/row,rowa1/ab2/b/row)
let $a := $row/a/text()
let $b := $row/b/text()
order by
  if ($sort-direction eq descending) then $row/*[fn:local-name(.) eq
$sort-node] else () descending empty least,
  if ($sort-direction ne descending) then $row/*[fn:local-name(.) eq
$sort-node] else () ascending empty greatest
return trtd{$a}/tdtd{$b}/td/tr


On Wed, Mar 27, 2013 at 1:28 PM, Tim t...@aaom.net wrote:

 Thanks Geert!

 ** **

 Tim

 ** **

 *From:* general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] *On Behalf Of *Geert Josten
 *Sent:* Wednesday, March 27, 2013 3:23 PM
 *To:* MarkLogic Developer Discussion
 *Subject:* Re: [MarkLogic Dev General] How to program the order by clause
 in FLWOR?

 ** **

 Hi Tim,

  

 Just leave out the braces, and replace $default with ()..

  

 J

  

 Cheers,

 Geert

  

 *Van:* general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] *Namens *Tim
 *Verzonden:* woensdag 27 maart 2013 20:19
 *Aan:* 'MarkLogic Developer Discussion'
 *Onderwerp:* [MarkLogic Dev General] How to program the order by clause
 in FLWOR?

  

 Hi Folks,

  

 Is there a simple way to program the order by clause in FLWOR to make it
 optional based on parameters?  I have a web page that I want to sort as
 indicated by the user.

  

 For example, this code illustrates what I’d like to do but doesn’t work:**
 **

  

 let $sort-option := “a descending”

 return

 for $row in $rows 

 let $a := $row/a/text()

 let $b := $row/b/text()

 order by *{if ($sort-option eq “a ascending”) then $a ascending else
 $default}*

 return trtd{$a}/tdtd{$b}/td/tr

  

 Thanks!

  

 Tim Meagher

  

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] gettext for XML and MarkLogic?

[Ryan Dew]

Those are interesting enhancements. I don't think it would be hard at all
to extend it to support injection of numbers or words. Simply have the map
entry contain an additional element that ties itself to a position in a
sequence of items passed and replace it appropriately.

To be honest, I have no idea how I would address singular/plural forms. It
seems like that would have to be handled externally and the bundle would
just need separate keys in a bundle for singular and plural forms. If you
have any suggestions, I would love to here them. Unless, you were just
suggesting, like the above, that you could inject a plural or singular form
of a word into a longer string, which is probably what you meant now that I
think about it.

Thanks,
Ryan Dew


On Thu, Feb 21, 2013 at 4:00 AM, Florent Georges li...@fgeorges.org wrote:

 Ryan Dew wrote:

   Hi,

  I'm sure it can be improved, but if you want to see an example
  of a library that makes use of map:map objects and server
  fields, like Mike mentioned, you can go here:
  https://github.com/ryanjdew/XQuery-i18n/blob/master/i18n.xqy

   Interesting.  But if I am right, it only resolves strings given
 two key: a name and a locale.  But the tricky part is to inject
 variable parts.  For instance numbers (computed by the app),
 adapting words with singular/plural forms, injecting additional
 words (e.g. adjectives generally don't go to the same place in EN
 and FR), etc.

   Did I miss anything?  Or do you have any plan to support some
 of those?

   Regards,

 --
 Florent Georges
 http://fgeorges.org/
 http://h2oconsulting.be/
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] gettext for XML and MarkLogic?

[Ryan Dew]

I like that approach. I opened an issue to look into it how I can
incorporate those ideas into my library. Who knows, though, Jakob might
just beat me to it.

Thanks,
Ryan Dew


On Thu, Feb 21, 2013 at 7:18 AM, Florent Georges li...@fgeorges.org wrote:

 Ryan Dew wrote:

  Those are interesting enhancements.  I don't think it would be
  hard at all to extend it to support injection of numbers or
  words. Simply have the map entry contain an additional element
  that ties itself to a position in a sequence of items passed
  and replace it appropriately.

   The problem is that the position itself could be dependent on
 the locale :-)

  To be honest, I have no idea how I would address singular/
  plural forms.  It seems like that would have to be handled
  externally and the bundle would just need separate keys in a
  bundle for singular and plural forms.  If you have any
  suggestions, I would love to here them.

   The approach I used a few years ago (God, that was in 2005
 actually, time's flying :-P) is described succently there:

 http://xsl.markmail.org/thread/zg4irqyap5begxmc

   Basically it uses XML to express a kind of format string,
 instead of simple strings, and use elements as placeholders.  It
 also had more sophisticated features for some domain-specific
 formatting, like formating a person using a format string, to
 have different address or name formatting.  But the general idea
 is something like:

 i18n xml:lang=en
l10n key=some.keyThe amount is amount/./l10n
 /i18n
 i18n xml:lang=fr
l10n key=some.keyLe montant est amount/./l10n
 /i18n

   The place where the text must appear (the caller) must
 provide a value for the place holder.  This value can itself be
 locale-dependent (e.g. different numbers formating, in EN you use
 . as the decimal point and in FR we use ,).  This example is
 very simple and could be resolved by using simple strings and by
 having the caller assembling them, but in some cases if you have
 2 variable parts A and B, in one language you will have part A
 appearing before B and in another language B appearing before A.
 With such a template format each localization can reorganize
 itself the variable parts.

   For variable parts that are themselves dependent on some
 dynamic values, the part can be a sort of a map:

 i18n xml:lang=en
l10n key=...The book contains number/
   sections sg=section pl=sections/./l10n
 /i18n
 i18n xml:lang=fr
l10n key=...Le livre contient number/
   sections sg=section pl=sections/./l10n
 /i18n

   The caller has to define that he'll pass a value for number
 (most likely a number) and a value for sections (either sg or
 pl to select the singular or plural form).  The result is a
 more-or-less readable sentence with elements for variable parts.

   This is just my recollection after several years, and the
 solution is probably not perfect, but it gives more flexibility
 to people localizing the keys, and developers internationalizing
 just have to define the different keys and the name of the
 variable parts if any.

   Regards,

 --
 Florent Georges
 http://fgeorges.org/
 http://h2oconsulting.be/

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] gettext for XML and MarkLogic?

[Ryan Dew]

Mine is more of a bundle concept. It looks like in the implementation Mike
mentioned that versions of text in the various languages are all stored in
the same XML document. In my code there are separate 'bundles' for each
language that look like the following:

i18n locale=en_US id=general
  map:map xmlns:map=http://marklogic.com/xdmp/map; xmlns:xsi=
http://www.w3.org/2001/XMLSchema-instance; xmlns:xs=
http://www.w3.org/2001/XMLSchema;
map:entry key=greeting
  map:value xsi:type=xs:stringhello/map:value
/map:entry
map:entry key=farewell
  map:value xsi:type=xs:stringgoodbye/map:value
/map:entry
  /map:map
/i18n

Even now looking at that, not sure why I didn't use the xml:lang attribute
instead locale. There is always something that can be tweaked...

-Ryan Dew


On Wed, Feb 20, 2013 at 2:52 PM, Jakob Fix jakob@gmail.com wrote:

 Thanks Michael and Ryan, I think this is what I'm looking for. Ryan, any
 chance to see a schema of the map:map object, is it inspired by the
 translate.xqy Michael mentioned? Or based on a bundle concept that is
 generally known (except by me ;-))?

 Looking forward to forking it!

 cheers,
 Jakob.



 On Tue, Feb 19, 2013 at 8:13 PM, Ryan Dew ryan.j@gmail.com wrote:

 I'm sure it can be improved, but if you want to see an example of a
 library that makes use of map:map objects and server fields, like Mike
 mentioned, you can go here:
 https://github.com/ryanjdew/XQuery-i18n/blob/master/i18n.xqy

 -Ryan Dew


 On Tue, Feb 19, 2013 at 10:58 AM, Michael Blakeley m...@blakeley.comwrote:

 Have you looked at /MarkLogic/appservices/utils/translate.xqy yet? It's
 undocumented and may not do everything you want, but it might offer some
 ideas.

 For a large lookup-table I would consider using map:map XML
 serialization. That way you can build the map quickly from stored XML, and
 lookups via map:get will be very fast. For even more speed you could cache
 the map in a server-field, with versioning to handle updates.

 -- Mike

 On 19 Feb 2013, at 07:16 , Jakob Fix jakob@gmail.com wrote:

  Hi,
 
  this may be slightly off-topic for the list  say, you want to
 maintain an application that provides several user interface languages
 (say, English, French and German), are there any best practices on managing
 labels and static text?
 
  I'm thinking of having a simple look-up XML file that contains the
 labels indexed by language. But then, the question is how to deal with
 plural forms, interspersing dynamic bits (like We have found  + 11 + 
 hits.) and all the other fun bits. I know that this is a general headache,
 and that there are solutions like gettext (and others?). Is there an
 implementation for XML, by any chance?
 
  Any experience with that in the context of MarkLogic?  Also, would a
 look-up XML be efficient when used via xquery and xslt?
 
  I'd be grateful for any pointers.
 
  cheers,
  Jakob.
  ___
  General mailing list
  General@developer.marklogic.com
  http://developer.marklogic.com/mailman/listinfo/general

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Collation Question

[Ryan Dew]

I think standard is the one I want. Very helpful, as always. Thank you!


On Wed, Feb 20, 2013 at 3:07 PM, Michael Blakeley m...@blakeley.com wrote:

 That could be a docs bug. Looking at 6.0-2.1 I see these options. Maybe
 'standard' is the one you want?

 zh-Hant@collation=big5han
 zh-Hant@collation=gb2312han
 zh-Hant@collation=pinyin
 zh-Hant@collation=standard

 -- Mike

 On 20 Feb 2013, at 13:33 , Ryan Dew ryan.j@gmail.com wrote:

  On the MarkLogic website under Encodings and Collations (
 http://docs.marklogic.com/guide/search-dev/encodings_collations#id_85649) the 
 following collation is listed for Chinese:
 http://marklogic.com/collation/zh-Hant@collation=stroke. However, if I
 try to use this collation on a server licensed for Chinese, I get the
 following.
 
  Code:
 
  declare default collation 
 http://marklogic.com/collation/zh-Hant@collation=stroke;;
  ()
 
  Error:
 
  XDMP-COLLATIONURI: (err:XQST0038) Collation URI
 http://marklogic.com/collation/zh-Hant@collation=stroke is not valid
 
  The other collations work, like German with the phonebook collation and
 Spanish with the traditional collation. Is there something I'm missing?
 
  Thanks,
  Ryan Dew
  ___
  General mailing list
  General@developer.marklogic.com
  http://developer.marklogic.com/mailman/listinfo/general

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Somebody please help me!

[Ryan Dew]

I believe this is because cts:uris is always unfiltered. You should get the
same result from cts:search result if you passed the 'unfiltered' option. I
believe you can get what you want if you set 'detail' as a fragment root.
You can read about fragment roots here:
http://docs.marklogic.com/guide/admin/fragments#id_34807

I maybe wrong since I've never tested this sort of thing with cts:uris, but
someone smarter than me might be able to have a more certain answer.

-Ryan Dew


On Tue, Feb 12, 2013 at 4:20 PM, Raghu raghupathy.visweswa...@gmail.comwrote:

 HI all,


 I have a parent - detail
 with two children -name , value
 I wrote an xquery to fetch the document with both name and value (should
 be siblings) with 'a' as value
 cts:search is working a s expected whereas cts:uris isn't
 Please find below the sample xquery
 Thanks in advance


 xdmp:document-insert(/1.xml,

 root

 detail

 namea/name

 valuea/value

 /detail

 detail

 namea/name

 valueb/value

 /detail

 /root

 ),

 xdmp:document-insert(/2.xml,

 root

 detail

  namea/name

 valueb/value

 /detail

 detail

 nameb/name

 valuea/value

 /detail

 /root

 ),

 xdmp:document-insert(/3.xml,

 root

  detail

 nameb/name

 valueb/value

 /detail

 detail

 nameb/name

 valueb/value

 /detail

 /root

 );

 (: both children of detail ‘name’ and ‘value’ should have ‘a’ as text
 content :)


 let $myquery :=  cts:element-query(xs:QName(detail),


   cts:and-query((



 cts:element-range-query(xs:QName(name),=,a),



 cts:element-range-query(xs:QName(value),=,a)


 ))


 )


 return


 (


 cts:uris/ cts:uri-match results:,


 cts:uris('',(),$myquery),


 cts:search results:,


 cts:search(/root,$myquery)


 )


 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Somebody please help me!

[Ryan Dew]

You can't have a filtered cts:uris call. You could do
cts:search(/root,$myquery)/fn:base-uri(), but that could be expensive
precisely because it is filtered. Just out of curiosity, though, I tried
setting up the fragment root locally and have confirmed that cts:uris works
as expected with the fragment root.

I'll let an expert speak to the memory impact, but my experience is that
there isn't much of an issue with root fragments. If you already have a lot
of data in your DB then it could take some time re-indexing, but that is
about all I've observed.

-Ryan Dew


On Tue, Feb 12, 2013 at 4:52 PM, Raghu raghupathy.visweswa...@gmail.comwrote:

 Introducing fragment roots is it gonna take additional memory/indexes?
 Is there an easier way without having to create any fragments/indexes?



 On Tue, Feb 12, 2013 at 6:48 PM, Raghu 
 raghupathy.visweswa...@gmail.comwrote:

 @Ryan - you are right cts:search is also giving the same results when
 unfiltered

 Is there an option to make cts:uris/ cts:uri-match behave like a filtered
 search??


 On Tue, Feb 12, 2013 at 6:31 PM, Ryan Dew ryan.j@gmail.com wrote:

 I believe this is because cts:uris is always unfiltered. You should get
 the same result from cts:search result if you passed the 'unfiltered'
 option. I believe you can get what you want if you set 'detail' as a
 fragment root. You can read about fragment roots here:
 http://docs.marklogic.com/guide/admin/fragments#id_34807

 I maybe wrong since I've never tested this sort of thing with cts:uris,
 but someone smarter than me might be able to have a more certain answer.

 -Ryan Dew


 On Tue, Feb 12, 2013 at 4:20 PM, Raghu raghupathy.visweswa...@gmail.com
  wrote:

 HI all,


 I have a parent - detail
 with two children -name , value
 I wrote an xquery to fetch the document with both name and value
 (should be siblings) with 'a' as value
 cts:search is working a s expected whereas cts:uris isn't
 Please find below the sample xquery
 Thanks in advance


 xdmp:document-insert(/1.xml,

 root

 detail

 namea/name

 valuea/value

 /detail

 detail

 namea/name

 valueb/value

 /detail

 /root

 ),

 xdmp:document-insert(/2.xml,

 root

 detail

  namea/name

 valueb/value

 /detail

 detail

 nameb/name

 valuea/value

 /detail

 /root

 ),

 xdmp:document-insert(/3.xml,

 root

  detail

 nameb/name

 valueb/value

 /detail

 detail

 nameb/name

 valueb/value

 /detail

 /root

 );

 (: both children of detail ‘name’ and ‘value’ should have ‘a’ as text
 content :)


 let $myquery :=  cts:element-query(xs:QName(detail),


   cts:and-query((



 cts:element-range-query(xs:QName(name),=,a),



 cts:element-range-query(xs:QName(value),=,a)


 ))


 )


 return


 (


 cts:uris/ cts:uri-match results:,


 cts:uris('',(),$myquery),


 cts:search results:,


 cts:search(/root,$myquery)


 )


 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general




 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Optimal strategy for finding duplicate values in database

[Ryan Dew]

Since you have a range index I believe you can do something like this:

cts:element-attribute-values(xs:QName('my:element'),xs:QName('my:attribute'))[cts:frequency(.)
gt 1]

You would need a second query to actually retrieve the documents with
duplicate ids, but still probably more efficient.

-Ryan Dew


On Fri, Feb 1, 2013 at 7:44 PM, David Sewell dsew...@virginia.edu wrote:

 Given a database with lots of files containing attribute values that are
 supposed to be unique across the database, is there an optimal
 MarkLogic-ish way to check for duplicates?

 One traditional approach to finding non-distinct values performs
 terribly:

for $value in distinct-values(collection()//my:element/@my:attribute)
return $value[count($values[. = $value])  1]

 (where by terribly I mean on the order of 10 seconds elapsed time for
 5000 values on my system). Leveraging an element-attribute range index
 and running cts:search() on the distinct values was somewhat better, but
 not enough.

 By far the most performant approach I have found is to iterate over a
 sorted sequence of values, simulating a Unix sort  file | uniq -d,
 like so:

 let $ordered_values :=
for $v in collection()//my:element/@my:attribute
order by $v
return $v
 for $val at $pos in $ordered_values
 return
   if ($val eq $ordered_values[$pos - 1])
   then $val
   else ()

 where performant means around 0.5 seconds for 10 values.

 Is this the best approach? Given that the attributes in question are in
 an element-attribute range index, is there another strategy worth
 trying?

 David

 --
 David Sewell, Editorial and Technical Manager
 ROTUNDA, The University of Virginia Press
 PO Box 400314, Charlottesville, VA 22904-4314 USA
 Email: dsew...@virginia.edu   Tel: +1 434 924 9973
 Web: http://rotunda.upress.virginia.edu/
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] fn:analyze-string() result non compliant

[Ryan Dew]

This has been around since at least version 4.2
http://docs.marklogic.com/4.2/fn:analyze-string?v=4.2. I know because I
wrote a blog post about it a while back.

Good point about the namespace, though. Perhaps the standard changed since
ML first implemented it?

-Ryan Dew
On Nov 10, 2012 5:09 AM, Florent Georges li...@fgeorges.org wrote:

   Hi,

   I am glad to see fn:analyze-string() is supported in MarkLogic
 version 6!  Just a difference with the current FO 3.0 [1]...
 When running:

 fn:analyze-string('ab', 'a')

 the result is:

 s:analyze-string-result xmlns:s=
 http://www.w3.org/2009/xpath-functions/analyze-string;
s:matcha/s:match
s:non-matchb/s:non-match
 /s:analyze-string-result

 but the namespace should be http://www.w3.org/2005/xpath-functions
 instead (see?, there is an extra /analyze-string at the end, and
 2009 should be 2005).

   Regards,

 --
 Florent Georges
 http://fgeorges.org/
 http://h2oconsulting.be/

 [1] http://w3.org/TR/xpath-functions-30/#func-analyze-string
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] XQuery 3.0

[Ryan Dew]

Not sure about 1,2,, and 4. But dynamic function calls are already in ML 6.

http://maxdewpoint.blogspot.com/2012/10/higher-order-functions-in-marklogic-6.html



On Mon, Oct 22, 2012 at 7:28 AM, David Sargeant da...@dsargeant.com wrote:

 I was just curious what the roadmap is for the following XQuery 3.0
 features:

 1. group by clause in FLWOR Expressions

 2.  allowing empty in for clause

 3. Dynamic function calls

 4. count clause in FLWOR Expressions


 Thanks,

 David

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

[MarkLogic Dev General] Map Operator and Concurrency

[Ryan Dew]

I know that flwor statements in certain situations can block concurrency.
I've done a couple simple tests that seem to indicate the map operator (!)
in ML6 doesn't break concurrency, but does anyone know for certain if there
is any difference with the map operator?


-Ryan Dew
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Map Operator and Concurrency

[Ryan Dew]

Sorry, I realized my test was flawed and the map operator does seem to
interfere with concurrency. Does anyone know if there is any benefit of
using the map operator over a flwor other than simpler syntax?

On Sat, Oct 20, 2012 at 5:22 PM, Ryan Dew ryan.j@gmail.com wrote:

 I know that flwor statements in certain situations can block concurrency.
 I've done a couple simple tests that seem to indicate the map operator (!)
 in ML6 doesn't break concurrency, but does anyone know for certain if there
 is any difference with the map operator?


 -Ryan Dew

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

[MarkLogic Dev General] Native Plugins on OSX 10.8.2

[Ryan Dew]

This past weekend I played around with creating my own native plugin. I was 
able to compile and package my plugin just fine, making use of the example make 
file provided. I also ran plugin:install-from-zip and it returned 1 as expected.

At this point, though, xdmp:host-status(xdmp:host())//*:native-plugins 
indicates that I don't have any native plugins installed and when I try to use 
my native plugin I get the following:

[1.0-ml] XDMP-NOLIBRARY: cts:aggregate(regex-audf, regex, 
cts:path-reference(/records/record/title,(type=string,collation=http://marklogic.com/collation/;)),
 Test .*8 Document) -- The shared library cannot be loaded: 
dlopen(/Users/ryandew/Library/Application 
Support/MarkLogic/Data/Lib/libregex-audf.dylib, 6)

The library file does exist at this location however. I tried restarting 
MarkLogic. Could it be a permissions issue?

Thanks in advance for any direction.

-Ryan Dew___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Native Plugins on OSX 10.8.2

[Ryan Dew]

It may also help that I tried these steps with the sample native plugin 
provided and got the same results. Here is the info that you were asking for:

Extensions DB:
/native/regex-audf/libregex-audf.dylib
/native/regex-audf/manifest.xml
/native/sampleplugin/libsampleplugin.dylib
/native/sampleplugin/manifest.xml 

find /Users/ryandew/Library/Application\ Support/MarkLogic/Data/Lib
/Users/ryandew/Library/Application Support/MarkLogic/Data/Lib
/Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/.DS_Store
/Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/Manifest
/Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/native
/Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/native/.DS_Store
/Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/native/regex-audf
/Users/ryandew/Library/Application 
Support/MarkLogic/Data/Lib/native/regex-audf/libregex-audf.dylib
/Users/ryandew/Library/Application 
Support/MarkLogic/Data/Lib/native/regex-audf/manifest.xml
/Users/ryandew/Library/Application 
Support/MarkLogic/Data/Lib/native/sampleplugin
/Users/ryandew/Library/Application 
Support/MarkLogic/Data/Lib/native/sampleplugin/libsampleplugin.dylib
/Users/ryandew/Library/Application 
Support/MarkLogic/Data/Lib/native/sampleplugin/manifest.xml


Manifest:
?xml version=1.0 encoding=UTF-8?
plugin xmlns=http://marklogic.com/extension/plugin;
  nameregex-audf/name
  idregex-audf/id
  version0.1/version
  provider-nameRyan Dew/provider-name
  descriptionNative plugin for working with regex/description
  native
pathlibregex-audf.dylib/path
  /native
/plugin

Thanks again for looking into this!

-Ryan Dew

On Sep 24, 2012, at 7:22 AM, John Snelson john.snel...@marklogic.com wrote:

 On 24/09/12 14:07, Ryan Dew wrote:
 This past weekend I played around with creating my own native plugin. I
 was able to compile and package my plugin just fine, making use of the
 example make file provided. I also ran plugin:install-from-zip and it
 returned 1 as expected.
 
 At this point, though, xdmp:host-status(xdmp:host())//*:native-plugins
 indicates that I don't have any native plugins installed
 
 That's expected. The host status only contains information for native 
 plugins that have been loaded. Native plugins are loaded lazily when 
 they are first used.
 
 and when I try to use my native plugin I get the following:
 
 
  [1.0-ml] XDMP-NOLIBRARY: cts:aggregate(regex-audf, regex,
  
 cts:path-reference(/records/record/title,(type=string,collation=http://marklogic.com/collation/;)),
  Test .*8 Document) -- The shared library cannot be loaded:
  dlopen(/Users/ryandew/Library/Application
  Support/MarkLogic/Data/Lib/libregex-audf.dylib, 6)
 
 
 The library file does exist at this location however. I tried restarting
 MarkLogic. Could it be a permissions issue?
 
 I'm not sure how, since MarkLogic saves the native plugins files out 
 into the Lib directory itself. I've successfully deployed and used a 
 native plugin to Mac OS X myself, so I'd suspect it might have more to 
 do with your plugin itself.
 
 Can you post a listing of the contents of your Extensions database?
 Also the result of the command find /Users/ryandew/Library/Application\ 
 Support/MarkLogic/Data/Lib or equivalent.
 
 Also does the library inside Lib look the same as the one you built 
 into a native plugin?
 
 Turning on debug logging should output more information about the native 
 plugin cache and plugin loading - that might help you work out what's 
 going on.
 
 John
 
 -- 
 John Snelson, Lead Engineerhttp://twitter.com/jpcs
 MarkLogic Corporation http://www.marklogic.com
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Native Plugins on OSX 10.8.2

[Ryan Dew]

Thanks. I''ll have to try that when I get home tonight.
On Sep 24, 2012 7:50 AM, John Snelson john.snel...@marklogic.com wrote:

 Try calling cts:aggregate() with a first argument of
 native/regex-audf. The scope you used when you installed the plugin
 (in this case native) needs to be included in the plugin name.

 John

 On 24/09/12 14:43, Ryan Dew wrote:
  It may also help that I tried these steps with the sample native plugin
 provided and got the same results. Here is the info that you were asking
 for:
 
  Extensions DB:
  /native/regex-audf/libregex-audf.dylib
  /native/regex-audf/manifest.xml
  /native/sampleplugin/libsampleplugin.dylib
  /native/sampleplugin/manifest.xml
 
  find /Users/ryandew/Library/Application\ Support/MarkLogic/Data/Lib
  /Users/ryandew/Library/Application Support/MarkLogic/Data/Lib
  /Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/.DS_Store
  /Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/Manifest
  /Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/native
  /Users/ryandew/Library/Application
 Support/MarkLogic/Data/Lib/native/.DS_Store
  /Users/ryandew/Library/Application
 Support/MarkLogic/Data/Lib/native/regex-audf
  /Users/ryandew/Library/Application
 Support/MarkLogic/Data/Lib/native/regex-audf/libregex-audf.dylib
  /Users/ryandew/Library/Application
 Support/MarkLogic/Data/Lib/native/regex-audf/manifest.xml
  /Users/ryandew/Library/Application
 Support/MarkLogic/Data/Lib/native/sampleplugin
  /Users/ryandew/Library/Application
 Support/MarkLogic/Data/Lib/native/sampleplugin/libsampleplugin.dylib
  /Users/ryandew/Library/Application
 Support/MarkLogic/Data/Lib/native/sampleplugin/manifest.xml
 
 
  Manifest:
  ?xml version=1.0 encoding=UTF-8?
  plugin xmlns=http://marklogic.com/extension/plugin;
 nameregex-audf/name
 idregex-audf/id
 version0.1/version
 provider-nameRyan Dew/provider-name
 descriptionNative plugin for working with regex/description
 native
   pathlibregex-audf.dylib/path
 /native
  /plugin
 
  Thanks again for looking into this!
 
  -Ryan Dew
 
  On Sep 24, 2012, at 7:22 AM, John Snelson john.snel...@marklogic.com
 wrote:
 
  On 24/09/12 14:07, Ryan Dew wrote:
  This past weekend I played around with creating my own native plugin. I
  was able to compile and package my plugin just fine, making use of the
  example make file provided. I also ran plugin:install-from-zip and it
  returned 1 as expected.
 
  At this point, though, xdmp:host-status(xdmp:host())//*:native-plugins
  indicates that I don't have any native plugins installed
 
  That's expected. The host status only contains information for native
  plugins that have been loaded. Native plugins are loaded lazily when
  they are first used.
 
  and when I try to use my native plugin I get the following:
 
 
[1.0-ml] XDMP-NOLIBRARY: cts:aggregate(regex-audf, regex,
 
 cts:path-reference(/records/record/title,(type=string,collation=
 http://marklogic.com/collation/;)),
Test .*8 Document) -- The shared library cannot be loaded:
dlopen(/Users/ryandew/Library/Application
Support/MarkLogic/Data/Lib/libregex-audf.dylib, 6)
 
 
  The library file does exist at this location however. I tried
 restarting
  MarkLogic. Could it be a permissions issue?
 
  I'm not sure how, since MarkLogic saves the native plugins files out
  into the Lib directory itself. I've successfully deployed and used a
  native plugin to Mac OS X myself, so I'd suspect it might have more to
  do with your plugin itself.
 
  Can you post a listing of the contents of your Extensions database?
  Also the result of the command find /Users/ryandew/Library/Application\
  Support/MarkLogic/Data/Lib or equivalent.
 
  Also does the library inside Lib look the same as the one you built
  into a native plugin?
 
  Turning on debug logging should output more information about the native
  plugin cache and plugin loading - that might help you work out what's
  going on.
 
  John
 
  --
  John Snelson, Lead Engineerhttp://twitter.com/jpcs
  MarkLogic Corporation http://www.marklogic.com
  ___
  General mailing list
  General@developer.marklogic.com
  http://developer.marklogic.com/mailman/listinfo/general
 
  ___
  General mailing list
  General@developer.marklogic.com
  http://developer.marklogic.com/mailman/listinfo/general
 


 --
 John Snelson, Lead Engineerhttp://twitter.com/jpcs
 MarkLogic Corporation http://www.marklogic.com
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Native Plugins on OSX 10.8.2

[Ryan Dew]

Just to follow up, that worked. I should have caught that in the documentation. 
Thanks again! 

-Ryan Dew

On Sep 24, 2012, at 9:04 AM, Ryan Dew ryan.j@gmail.com wrote:

 Thanks. I''ll have to try that when I get home tonight.
 
 On Sep 24, 2012 7:50 AM, John Snelson john.snel...@marklogic.com wrote:
 Try calling cts:aggregate() with a first argument of
 native/regex-audf. The scope you used when you installed the plugin
 (in this case native) needs to be included in the plugin name.
 
 John
 
 On 24/09/12 14:43, Ryan Dew wrote:
  It may also help that I tried these steps with the sample native plugin 
  provided and got the same results. Here is the info that you were asking 
  for:
 
  Extensions DB:
  /native/regex-audf/libregex-audf.dylib
  /native/regex-audf/manifest.xml
  /native/sampleplugin/libsampleplugin.dylib
  /native/sampleplugin/manifest.xml
 
  find /Users/ryandew/Library/Application\ Support/MarkLogic/Data/Lib
  /Users/ryandew/Library/Application Support/MarkLogic/Data/Lib
  /Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/.DS_Store
  /Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/Manifest
  /Users/ryandew/Library/Application Support/MarkLogic/Data/Lib/native
  /Users/ryandew/Library/Application 
  Support/MarkLogic/Data/Lib/native/.DS_Store
  /Users/ryandew/Library/Application 
  Support/MarkLogic/Data/Lib/native/regex-audf
  /Users/ryandew/Library/Application 
  Support/MarkLogic/Data/Lib/native/regex-audf/libregex-audf.dylib
  /Users/ryandew/Library/Application 
  Support/MarkLogic/Data/Lib/native/regex-audf/manifest.xml
  /Users/ryandew/Library/Application 
  Support/MarkLogic/Data/Lib/native/sampleplugin
  /Users/ryandew/Library/Application 
  Support/MarkLogic/Data/Lib/native/sampleplugin/libsampleplugin.dylib
  /Users/ryandew/Library/Application 
  Support/MarkLogic/Data/Lib/native/sampleplugin/manifest.xml
 
 
  Manifest:
  ?xml version=1.0 encoding=UTF-8?
  plugin xmlns=http://marklogic.com/extension/plugin;
 nameregex-audf/name
 idregex-audf/id
 version0.1/version
 provider-nameRyan Dew/provider-name
 descriptionNative plugin for working with regex/description
 native
   pathlibregex-audf.dylib/path
 /native
  /plugin
 
  Thanks again for looking into this!
 
  -Ryan Dew
 
  On Sep 24, 2012, at 7:22 AM, John Snelson john.snel...@marklogic.com 
  wrote:
 
  On 24/09/12 14:07, Ryan Dew wrote:
  This past weekend I played around with creating my own native plugin. I
  was able to compile and package my plugin just fine, making use of the
  example make file provided. I also ran plugin:install-from-zip and it
  returned 1 as expected.
 
  At this point, though, xdmp:host-status(xdmp:host())//*:native-plugins
  indicates that I don't have any native plugins installed
 
  That's expected. The host status only contains information for native
  plugins that have been loaded. Native plugins are loaded lazily when
  they are first used.
 
  and when I try to use my native plugin I get the following:
 
 
[1.0-ml] XDMP-NOLIBRARY: cts:aggregate(regex-audf, regex,

  cts:path-reference(/records/record/title,(type=string,collation=http://marklogic.com/collation/;)),
Test .*8 Document) -- The shared library cannot be loaded:
dlopen(/Users/ryandew/Library/Application
Support/MarkLogic/Data/Lib/libregex-audf.dylib, 6)
 
 
  The library file does exist at this location however. I tried restarting
  MarkLogic. Could it be a permissions issue?
 
  I'm not sure how, since MarkLogic saves the native plugins files out
  into the Lib directory itself. I've successfully deployed and used a
  native plugin to Mac OS X myself, so I'd suspect it might have more to
  do with your plugin itself.
 
  Can you post a listing of the contents of your Extensions database?
  Also the result of the command find /Users/ryandew/Library/Application\
  Support/MarkLogic/Data/Lib or equivalent.
 
  Also does the library inside Lib look the same as the one you built
  into a native plugin?
 
  Turning on debug logging should output more information about the native
  plugin cache and plugin loading - that might help you work out what's
  going on.
 
  John
 
  --
  John Snelson, Lead Engineerhttp://twitter.com/jpcs
  MarkLogic Corporation http://www.marklogic.com
  ___
  General mailing list
  General@developer.marklogic.com
  http://developer.marklogic.com/mailman/listinfo/general
 
  ___
  General mailing list
  General@developer.marklogic.com
  http://developer.marklogic.com/mailman/listinfo/general
 
 
 
 --
 John Snelson, Lead Engineerhttp://twitter.com/jpcs
 MarkLogic Corporation http://www.marklogic.com
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com

Re: [MarkLogic Dev General] We Got Six

[Ryan Dew]

Looks like a great release! I noticed that there are also some less
pronounced, but equally cool, updates which I've mentioned on my blog (
http://maxdewpoint.blogspot.com/2012/09/marklogic-60-released.html ). Is
there a reason some of these updates aren't mentioned or perhaps I'm just
missing where the updates were announced?

-Ryan Dew

On Mon, Sep 17, 2012 at 7:07 AM, Eric Bloch eric.bl...@marklogic.comwrote:

 See http://developer.marklogic.com/blog/we-got-six
 *
 *
 *We Got Six*

 Today, we at MarkLogic are very proud to release MarkLogic 
 6http://developer.marklogic.com/products/marklogic-server/6.0.
  It's more powerful, more accessible, and trusted as ever.   Below are some
 highlights

 *New APIs!*

 We've created entire new RESThttp://developer.marklogic.com/try/rest/index
  and Java http://developer.marklogic.com/try/java/index APIs to make
 the awesome that is MarkLogic available to vast new communities of software
 developers.  These APIs enable developers familiar with Java or REST APIs
 to get going quickly, yet they are also full-featured.  Their rich
 functionality enables developers to build complete database and search
 applications.

 Beyond the product documentation (guides and references) as well as new
 tutorials http://developer.marklogic.com/learn we've provided to get
 folks going, there is also a sample application built entirely via the
 MarkLogic Java API here http://developer.marklogic.com/code/top-songs.


 *Now with JSON*
 *
 *

 Along with these new APIs, you'll also find support for 
 JSONhttp://json.org/.
  The *data format of the web* is now easily stored, searched, queried,
 transformed, and analyzed directly in MarkLogic.


 *In-Database* *Analytic Functions and** MapReduce *
 *
 *

 Performance is a constant focus here and this feature is all about that.
  MarkLogic 6 comes with a large set of analytic functions built on top of
 our internal mechanism for distributed processing (aka 
 MapReducehttp://en.wikipedia.org/wiki/MapReduce)
 to provide high-performance calculations. Beyond these tools, for
 developers who need to push the envelope, we are also exposing the APIs as
 well as plugin interfaces to the MapReduce mechanism inside MarkLogic.
 Seriously. You can provide your own C/C++ functions for computing custom
 analytics that MarkLogic will run close to your data.


 *BI Tools Inteface*
 *
 *

 You can hook up IBM Cognoshttp://www-01.ibm.com/software/analytics/cognos/
  or Tableau http://www.tableausoftware.com/ to MarkLogic via a live
 connection for real-time reporting and analytic investigations. That in
 itself is cool. But the mechanism we're providing for this is... an ODBC
 driver. For now, the driver is only supported for use with Cognos and
 Tableau, but you can imagine what we're thinking here with this awesome
 first step. Yep, a SQL interface to a NoSQL database. Interesting.

 *
 *
 *UI Widgets*


 The MarkLogic Visualization 
 Widgetshttp://developer.marklogic.com/products/visualization-widgets are
 a set of open-source-licensed HTML5 widgets for charts, graphs and maps
 that provide out-of-the box user interface connectivity to MarkLogic. We
 provide the widgets, of course, but we've also added them into Application
 Builder, so that you can, with a few quick keystrokes and clicks, build an
 advanced search application with these visualization tools.


 *Content Pump*
 *
 *

 With this new command-line toolhttp://developer.marklogic.com/products/mlcp,
 it's a snap to load content from an OS filesystem or Hadoops 
 HDFShttp://hadoop.apache.org/index.html to
 MarkLogic.  You can also just as easily export from MarkLogic to a
 filesystem or HDFS.  The tool also introduces a new platform-independent
 MarkLogic database archive format.

 *Documentation*

 What about the docs http://docs.marklogic.com/?  Every developer knows
 how important good documenation is.

 Since I took over the helm of http://developer.marklogic.com in 2010,
 there are two things I've wanted to improve in this area. One is to make it
 easier for folks to contribute  - and the other is to improve site search.
 Both of these are still works in progress, but as we launch MarkLogic 6,
 we've made real advances in these areas. In particular, we've integrated
 documentation browsing and search directly into the MarkLogic application
 that runs our site, enabling Disqus http://www.disqus.com/ comments
 directly in our reference pages.  We now have excellent URLs for our docs,
 a reasonable HTML representation of our content, and integrated search and
 faceting. The feature list for the integrated docs.marklogic.com site
 includes:


- Product documentation is integrated seamlessly in the developer
   site. This includes MarkLogic 6 docs (as well as older versions for
   customers who've not yet upgraded)
   - All the content is *searchable via MarkLogic search* (including search
   suggestions, facets and snippeted results)
   - We have an *interactive

Re: [MarkLogic Dev General] ctx and xdmp declaration

[Ryan Dew]

The 1.0 version of the query doesn't run or the debugger is indicating an 
issue? If the query is not executing than it may be a bug in the version of ML 
you are using. In ML version 5.0-3.3 I'm able to run the following in qconsole 
just fine:

xquery version 1.0;

declare namespace xdmp = http://marklogic.com/xdmp;;
declare namespace cts = http://marklogic.com/cts;;

xdmp:unquote('test/','http://developer.envisn.com/xmlns/envisn/netvisn/p')

If the issue you are having is with the debugger, I don't know how to help you 
there, unfortunately.

-Ryan Dew


On Sep 14, 2012, at 7:08 AM, Gary Larsen gary.lar...@envisn.com wrote:

 Thanks Ryan.  That worked but now I’ve run into a function signature issue.
  
 In the xdmp:unquote() call below I get this syntax error in Oxygen’s debugger:
  
  Cannot find a matching 2-argument function named 
  {http://marklogic.com/xdmp}unquote()
  
 declare function console-data:param($name, $val) {
 let $p 
 :=xdmp:unquote($params,'http://developer.envisn.com/xmlns/envisn/netvisn/p')/p:params/*[local-name()=$name]/p:val
  return
 if(fn:count($p) gt 1 or $p ne '') then $p
 else $val
 };
 
  
 The query runs when the xquery version is “1.0.-ml” but the debugger requires 
 “1.0”.
  
 Is there a solution or alternative anyone can suggest?
  
 Thanks,
 Gary
  
 From: general-boun...@developer.marklogic.com 
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of Ryan Dew
 Sent: Thursday, September 13, 2012 5:23 PM
 To: MarkLogic Developer Discussion
 Subject: Re: [MarkLogic Dev General] ctx and xdmp declaration
  
 declare namespace xdmp = http://marklogic.com/xdmp;;
 declare namespace cts = http://marklogic.com/cts;;
  
 The namespaces could be found with running the following in QConsole:
  
 xquery version 1.0-ml;
  
 fn:namespace-uri-from-QName(xs:QName('xdmp:test')),
 fn:namespace-uri-from-QName(xs:QName('cts:test'))
  
  
 -Ryan Dew
  
 On Thu, Sep 13, 2012 at 3:08 PM, Gary Larsen gary.lar...@envisn.com wrote:
 Hi,
  
 Trying to debug a query in Oxygen but have syntax errors:
  
 XQuery syntax error in #...ns/envisn/netvisn/p')/p:params#: Prefix xdmp 
 has not been declared
 XQuery syntax error in #... cts:element-value-query#: Prefix cts has not 
 been declared
 
 What are the proper namespace declarations for these prefixes?
  
 Thanks,
 Gary
  
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
  
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Can one declare optional external variables?

[Ryan Dew]

I believe you can do something like this:

declare variable $var1-external external;
declare variable $var1 := try {$var1-external} catch($e) {()};

I will be nice once XQuery 3.0 is supported because then you could have a
default value for external variables like so:

declare variable $var1-external external := ();

-Ryan Dew


2012/9/13 Geert Josten geert.jos...@dayon.nl

 Hi,

 Is it possible to declare an external variable in such a way that you can
 still call the .xqy directly, without specifying that variable?

 Kind regards,
 Geert

 M.Sc. G.P.H. (Geert) Josten
 Senior Developer


 Dayon B.V.
 Delftechpark 37b
 2628 XJ Delft
 The Netherlands

 T +31 (0)88 26 82 570

 geert.jos...@dayon.nl
 www.dayon.nl

 De informatie - verzonden in of met dit e-mailbericht - is afkomstig van
 Dayon BV en is uitsluitend bestemd voor de geadresseerde. Indien u dit
 bericht onbedoeld hebt ontvangen, verzoeken wij u het te verwijderen. Aan
 dit bericht kunnen geen rechten worden ontleend.
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] ctx and xdmp declaration

[Ryan Dew]

declare namespace xdmp = http://marklogic.com/xdmp;;
declare namespace cts = http://marklogic.com/cts;;

The namespaces could be found with running the following in QConsole:

xquery version 1.0-ml;

fn:namespace-uri-from-QName(xs:QName('xdmp:test')),
fn:namespace-uri-from-QName(xs:QName('cts:test'))


-Ryan Dew

On Thu, Sep 13, 2012 at 3:08 PM, Gary Larsen gary.lar...@envisn.com wrote:

 Hi,

 ** **

 Trying to debug a query in Oxygen but have syntax errors:

 ** **

 XQuery syntax error in #...ns/envisn/netvisn/p')/p:params#: Prefix
 xdmp has not been declared

 XQuery syntax error in #... cts:element-value-query#: Prefix cts has
 not been declared

 

 What are the proper namespace declarations for these prefixes?

 ** **

 Thanks,

 Gary

 ** **

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Performance of fn:exists(fn:doc($uri))

[Ryan Dew]

It seems like it would be better to focus on using something more likely to be 
unique than xdmp:random rather than focusing on read locks. 

http://maxdewpoint.blogspot.com/2012/08/generate-unique-ids-for-collision.html

-Ryan Dew 

On Aug 29, 2012, at 11:25 PM, Geert Josten geert.jos...@dayon.nl wrote:

 Hi Mike,
 
 Not quite sure, but the conflict occurs when the uri doesn't exist yet, so
 there would be nothing to lock. Does that still create a read-lock?
 
 And in case the uri does exist, wouldn't this create potentially a lot of
 unnecessary read-locks (in case it takes a lot of attempts to find an
 unused uri)?
 
 Kind regards,
 Geert
 
 -Oorspronkelijk bericht-
 Van: general-boun...@developer.marklogic.com
 [mailto:general-boun...@developer.marklogic.com] Namens Michael Blakeley
 Verzonden: woensdag 29 augustus 2012 21:35
 Aan: MarkLogic Developer Discussion
 Onderwerp: Re: [MarkLogic Dev General] Performance of
 fn:exists(fn:doc($uri))
 
 No, you can't do that safely because cts:uris-match won't take a
 read-lock. You are opening yourself up to a race condition. And in some
 circumstances it will be slower than the recommended technique. There
 seems to be a popular idea that cts:uris-match() is always fastest, but
 that is not always true.
 
 The recommended technique is probably the fastest way to guarantee a new,
 unique URI. If you are going through the process of inserting a new
 document, this technique adds very little extra work. The document-insert
 itself always has to look for an existing document, because it might be
 replacing an existing document or it might be inserting a new document. It
 always has to write-lock the URI. So the extra exists() call merely
 repeats the URI lookup, which is cheap because it will be cached for the
 xdmp:document-insert call, and also gets a read-lock before
 xdmp:document-insert gets the write lock. In the vanishingly rare event
 that xdmp:random() produces an existing URI, this extra work is repeated -
 but is still quite cheap.
 
 -- Mike
 
 On 29 Aug 2012, at 12:29 , William Merritt Sawyer wrote:
 
 If you have the uri-lexicon turned on you can use
 cts:uri-match(fn:concat(/document-, xdmp:random(), .xml))
 
 From: general-boun...@developer.marklogic.com
 [mailto:general-boun...@developer.marklogic.com] On Behalf OfDanny Sinang
 Sent: Wednesday, August 29, 2012 12:33 PM
 To: MarkLogic Developer Discussion
 Subject: Re: [MarkLogic Dev General] Performance of
 fn:exists(fn:doc($uri))
 
 Thanks Geert.
 
 I did try fn:exists(fn:doc($uri))  on CQ before your response came in
 and found it to be fast.
 
 The locking / prevention of duplicate id's is discussed in
 http://markmail.org/message/mm5vtacpdzwfy44j  .
 
 Regards,
 Danny
 
 On Wed, Aug 29, 2012 at 2:23 PM, Geert Josten geert.jos...@dayon.nl
 wrote:
 Hi Danny,
 
 Performance should be easy to measure. Call the function from within
 QConsole x number of time and request profile output. Do the same while
 using xdmp:exists instead of fn:exists. That function works only on
 (partially) searchable expression, because it doesn't retrieve the actual
 content. It won't create a read-lock either, but I'm not sure why you want
 one. It won't prevent duplicate id's from being generated in concurrent
 requests..
 
 Kind regards,
 Geert
 
 Van: general-boun...@developer.marklogic.com
 [mailto:general-boun...@developer.marklogic.com] Namens Danny Sinang
 Verzonden: woensdag 29 augustus 2012 19:11
 Aan: general
 Onderwerp: [MarkLogic Dev General] Performance of
 fn:exists(fn:doc($uri))
 
 Hi,
 
 ML support suggested we do this to generate a unique ID for our
 documents :
 
 declare function choose-uri() as xs:string
{
   let $uri := fn:concat(/document-, xdmp:random(), .xml)
   return if (fn:exists(fn:doc($uri))) then choose-uri() else $uri
};
 
 My question is, will the call to fn:exists(fn:doc($uri)) be fast,
 considering that we now have 8 million documents ?
 
 The fn:exists(fn:doc($uri)) call is needed to obtain a read lock, which
 will be upgraded to a write lock when xdmp:document-insert is called.
 
 Regards,
 Danny
 
 
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Performance of fn:exists(fn:doc($uri))

[Ryan Dew]

You could use fn:doc-available and that doesn't acrually fetch the document
or require the uri lexicon.
On Aug 29, 2012 1:29 PM, William Merritt Sawyer 
william.saw...@ldschurch.org wrote:

  If you have the uri-lexicon turned on you can use
 cts:uri-match(fn:concat(/document-, xdmp:random(), .xml)) 

 ** **

 *From:* general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] *On Behalf Of *Danny Sinang
 *Sent:* Wednesday, August 29, 2012 12:33 PM
 *To:* MarkLogic Developer Discussion
 *Subject:* Re: [MarkLogic Dev General] Performance of
 fn:exists(fn:doc($uri))

 ** **

 Thanks Geert.

 ** **

 I did try fn:exists(fn:doc($uri))  on CQ before your response came in and
 found it to be fast.

 ** **

 The locking / prevention of duplicate id's is discussed in
 http://markmail.org/message/mm5vtacpdzwfy44j  .

 ** **

 Regards,

 Danny

 On Wed, Aug 29, 2012 at 2:23 PM, Geert Josten geert.jos...@dayon.nl
 wrote:

 Hi Danny,

  

 Performance should be easy to measure. Call the function from within
 QConsole x number of time and request profile output. Do the same while
 using xdmp:exists instead of fn:exists. That function works only on
 (partially) searchable expression, because it doesn’t retrieve the actual
 content. It won’t create a read-lock either, but I’m not sure why you want
 one. It won’t prevent duplicate id’s from being generated in concurrent
 requests..

  

 Kind regards,

 Geert

  

 *Van:* general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] *Namens *Danny Sinang
 *Verzonden:* woensdag 29 augustus 2012 19:11
 *Aan:* general
 *Onderwerp:* [MarkLogic Dev General] Performance of
 fn:exists(fn:doc($uri))

  

 Hi,

  

 ML support suggested we do this to generate a unique ID for our documents :
 

  

 declare function choose-uri() as xs:string
 {
let $uri := fn:concat(/document-, xdmp:random(), .xml)
return if (fn:exists(fn:doc($uri))) then choose-uri() else $uri
 }; 

   

 My question is, will the call to fn:exists(fn:doc($uri)) be fast,
 considering that we now have 8 million documents ?

  

 The fn:exists(fn:doc($uri)) call is needed to obtain a read lock, which
 will be upgraded to a write lock when xdmp:document-insert is called.

  

 Regards,

 Danny

  

  


 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

 ** **

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] sorting by date

[Ryan Dew]

You could do something like:

order by if ($doc/sort-date castable as xs:date) then xs:date($doc/sort-date) 
else () empty greatest

As for the indexing part, somebody more knowledgable will have to step in to 
answer that affirmatively, but I don't think it would hurt to try the following 
in qconsole if you already have the data and the indexes setup:

order by $doc/sort-date empty greatest

-Ryan Dew

On Jul 12, 2012, at 6:29 AM, Michael Sokolov wrote:

 I have a date-valued element range index.  I want to use it for 
 sorting.  When I do that using an expression like this:
 
 order by xs:date($doc//sort-date)
 
 I get errors when the $doc sequence contains documents that have no 
 sort-date.
 
 What is the recommended approach to handling nulls when sorting? Ideally 
 we would want them to sort last.
 
 If I just use
 
 order by $doc//sort-date
 
 will the index come in to play?
 
 -Mike Sokolov
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] sorting by date

[Ryan Dew]

That actually makes sense to to me since the empty values wouldn't be in
the index. if you need it so it includes empty and shows them first you
might want to try separating the queries, so one query returning the
documents without sortdate and a second query returning the items ascending
like:

(
for $doc in cts:search(fn:collection(),
 cts:and-query((
 cts:directory-query (/product/),

 cts:not-query(cts:element-query(fn:QName(,sortdate),cts:and-query((
))
 )
return ... ,
for $doc in xdmp:directory (/product/)[.//sortdate]
order by $doc//sortdate ascending
return ...
)

On Thu, Jul 12, 2012 at 8:11 AM, Mike Sokolov soko...@ifactory.com wrote:

 Thanks for the response, Geert and Ryan.  Actually it turns out that
 there is some magic afoot, because the cast is not actually needed.  I
 think it used to be in older MarkLogic versions.

 xdmp:query-trace(true()),
 (
 for $doc in xdmp:directory (/product/)
 order by $doc//sortdate descending empty least
 return ...
 )[1 to 20]

 runs fast, with the index (23 msec as opposed to 9+ sec when the 50,000
 docs need to be scanned)

 However, interestingly, if you change the order (ascending instead of
 descending) so that the empty items sort first, then the index is not
 used, and the query is even slower (14+ seconds)

 On 07/12/2012 08:38 AM, Geert Josten wrote:
  Hi Mike,
 
  Did you try:
 
  order by $doc//sort-date/xs:date(.)
 
  ?
 
  Kind regards,
  Geert
 
 
  -Oorspronkelijk bericht-
  Van: general-boun...@developer.marklogic.com [mailto:general-
  boun...@developer.marklogic.com] Namens Michael Sokolov
  Verzonden: donderdag 12 juli 2012 14:29
  Aan: MarkLogic Developer Discussion
  Onderwerp: [MarkLogic Dev General] sorting by date
 
  I have a date-valued element range index.  I want to use it for
  sorting.  When I do that using an expression like this:
 
  order by xs:date($doc//sort-date)
 
  I get errors when the $doc sequence contains documents that have no
  sort-date.
 
  What is the recommended approach to handling nulls when sorting? Ideally
  we would want them to sort last.
 
  If I just use
 
  order by $doc//sort-date
 
  will the index come in to play?
 
  -Mike Sokolov
  ___
  General mailing list
  General@developer.marklogic.com
  http://developer.marklogic.com/mailman/listinfo/general
 
  ___
  General mailing list
  General@developer.marklogic.com
  http://developer.marklogic.com/mailman/listinfo/general
 
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Listing Directories?

[Ryan Dew]

You could try something like:

cts:uris((),properties, cts:properties-query(

cts:element-query(xs:QName(prop:directory),cts:and-query(()))
))

-Ryan Dew

On Thu, Jul 5, 2012 at 12:27 PM, Alex Milowski a...@milowski.com wrote:

 I'm curious as to how certain directories get created in my database.  I
 have:

23,135,073 documents
9,916 directories

 I don't explicitly create directories in any of my import pipelines.

 If I could list them (or some sampling of them), I might have a better
 idea of where they are coming from.

 In the documentation, there is an example:

for $x in xdmp:document-properties()/prop:properties/prop:directory
   return directory-uri{xdmp:node-uri($x)}/directory-uri

 but that is a terrible idea if you have millions of documents.

 Is there a way to get the directories directly somehow?

 --
 --Alex Milowski
 The excellence of grammar as a guide is proportional to the paucity of the
 inflexions, i.e. to the degree of analysis effected by the language
 considered.

 Bertrand Russell in a footnote of Principles of Mathematics
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Listing Directories?

[Ryan Dew]

So why is the example from the documentation so bad? The expression is
fully searchable and will only return directory properties, so having
millions of documents shouldn't matter. The only count that should really
matter are the number of directories. Then if you really wanted to you
could paginate over the results using fn:subsequence.

xdmp:plan(xdmp:document-properties()/prop:properties/prop:directory)

and this might be slightly more efficient:

xdmp:plan(xdmp:document-properties()/prop:properties[prop:directory])

On Thu, Jul 5, 2012 at 12:49 PM, Will Thompson
wthomp...@jonesmcclure.comwrote:

 Do you have directory creation set to automatic in your database settings?

 -Will

 -Original Message-
 From: general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] On Behalf Of Alex Milowski
 Sent: Thursday, July 05, 2012 11:48 AM
 To: MarkLogic Developer Discussion
 Subject: Re: [MarkLogic Dev General] Listing Directories?

 On Thu, Jul 5, 2012 at 11:37 AM, Danny Sokolsky 
 danny.sokol...@marklogic.com wrote:
  You don't need the URI lexicon for the xdmp:estimate one.

 OK.  Good to know.

 What I really want to know is the name and purpose of these directories
 since I don't actually create them explicitly.

 --
 --Alex Milowski
 The excellence of grammar as a guide is proportional to the paucity of
 the inflexions, i.e. to the degree of analysis effected by the language
 considered.

 Bertrand Russell in a footnote of Principles of Mathematics
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general
 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Marklogic support for stemming

[Ryan Dew]

You can see and add your own entries using the custom dictionary library,
but you can't see or modify anything but your custom changes. The stemming
values that MarkLogic ships with you can't see or modify. I would have to
test to see if the custom dictionary overrides MarkLogic's dictionary that
they ship with.

-Ryan Dew

On Thu, Jun 14, 2012 at 8:07 AM, manoj viswanadha 
manoj.viswana...@gmail.com wrote:

 Hello all,

 I am using marklogic stemming enabled for databases. I got to know that ML
 stemming uses custom dictionary for stemming searches.
 While trying to get the xml file in the custom dictionary using *
 cdict:dictionary-read(en)* i am getting empty sequence, its not
 returning the xml
 file. But wen i triedsearch uisng stemmed enabled  its working fine.

 My requirement is that if that xml file is returned i want to edit that
 and add my own stems and keys for that file. How can i get that xml file
 returned?

 I am using Marklogic-5 free version.

 Thanks,
 Manoj

 ___
 General mailing list
 General@developer.marklogic.com
 http://community.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://community.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Presta - code management and parallel execution

[Ryan Dew]

Just poking through the code, this looks really awesome. Thanks for sharing! I 
am already brainstorming cool ways to make use of this.

One comment, though. Do you really want to move cprof into Presta? It seems 
that cprof fills quite a different purpose than Presta (despite working with 
the same xdmp functions), unless I'm missing something. Have you considered 
including cprof into Presta as a git submodule?

-Ryan Dew

On Jun 1, 2012, at 6:20 PM, Michael Blakeley wrote:

 https://github.com/mblakele/presta is now available. This library started 
 when I noticed the new 'result' option for xdmp:spawn in MarkLogic 5.0. That 
 looked handy, but xdmp:spawn still wanted me to supply a module path. I 
 wanted to write XQuery tasks with arbitrary code strings, like this:
 
for $i in 1 to 4
return xdmp:spawn-eval(
  'xdmp:sleep(1000)', (),
  options xmlns=xdmp:eval
resulttrue/result
  /options),
xdmp:elapsed-time()
 
 ...and have all four tasks execute in parallel, using all my CPUs and 
 returning the results. Here's what I ended up with:
 
import module namespace presta=com.blakeley.presta
  at presta.xqy;
 
let $presta-id := presta:prepare('xdmp:sleep(1000)')
for $i in 1 to 4
return presta:spawn(
  $presta-id, (),
  options xmlns=xdmp:eval
resulttrue/result
  /options),
xdmp:elapsed-time()
 
 The presta:spawn function works just like xdmp:spawn, and takes the same 
 arguments - except that the first parameter is a presta id not a module path. 
 We get that id by calling presta:prepare, which takes the XQuery string and 
 stores it the Modules database.
 
 The id for a Presta module is based on xdmp:hash64, so preparing the same 
 XQuery twice will return the same id. Presta tries not to update the Modules 
 database any more than necessary - but it's still best to stash these ids 
 somewhere if you can.
 
 Once prepared, we can also use a presta id with presta:invoke. This is handy 
 for apps that eval the same strings frequently,  because the Presta module 
 works with the module cache. And we don't have to keep the XQuery around 
 after it has been registered. This could be a way for developers using Java 
 to get some of the performance benefits of invoke, without having to write 
 every XQuery module in advance.
 
 But there's more to MarkLogic than XQuery. We can also call presta:prepare 
 with XSLT, and then call presta:xslt-invoke.
 
 We need libraries, too. So you can call presta:import to install a library 
 module where other Presta functions can find it.
 
presta:import(
  'lib-test.xqy',
'module namespace t=com.blakeley.presta.test;
 declare function t:now() { fn:current-dateTime() };')
 
presta:invoke(
  p:prepare(
'import module namespace t=com.blakeley.presta.test
   at lib-test.xqy;
 t:now()
 
 I almost forgot about environments with multiple applications. Every 
 app-server automatically gets its own Presta environment, based on the 
 appserver id from xdmp:server(). If you want multiple servers to share the 
 same Presta modules, we can call presta:appkey-set('my-app-key'). If you want 
 to keep modules somewhere other than the Modules database, set the app-server 
 modules location to any database you like.
 
 Of course I also wanted Presta to work with profiling. Presta supports cprof, 
 and includes a copy of the cprof library. My plan is to discontinue cprof as 
 a separate project, and include it as part of Presta.
 
 Finally, Presta works with the security model. Presta installs a new role 
 that you can grant to any user.
 
 The code is at https://github.com/mblakele/presta - I hope it's useful.
 
 -- Mike
 ___
 General mailing list
 General@developer.marklogic.com
 http://community.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://community.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Presta - code management and parallel execution

[Ryan Dew]

No real problems, just in the case of using just cprof there would be extra 
dead code. In extreme cases, a larger Modules database could cause the 
invalidating and refreshing of the Modules cache to take longer. Certainly, 
there are bigger issues if it comes to the point of worrying about a couple 
extra xqy files.

I agree it would be nice to have some sort of dependency management.

-Ryan

On Jun 2, 2012, at 11:57 AM, Michael Blakeley wrote:

 Thanks for mentioning submodules, since I was unaware of that option. They 
 look useful, but perhaps a bit awkward for this situation. I might look into 
 a subtree merge strategy, or the git-subtree code from Avery Pennarun.
 
 Tabling submodules and related approaches for a moment... what problems do 
 you see with rolling cprof directly into Presta? They are still separate 
 library modules, so a developer who only wants the cprof portion can skip the 
 presta:install step and use cprof independently. Perhaps I should separate 
 the test cases too, just to make life easier for cprof-only developers.
 
 In an ideal world there would be an XQuery package installation system with 
 dependency management. Meanwhile I am trying for the least evil solution.
 
 -- Mike
 
 On 2 Jun 2012, at 08:17 , Ryan Dew wrote:
 
 Just poking through the code, this looks really awesome. Thanks for sharing! 
 I am already brainstorming cool ways to make use of this.
 
 One comment, though. Do you really want to move cprof into Presta? It seems 
 that cprof fills quite a different purpose than Presta (despite working with 
 the same xdmp functions), unless I'm missing something. Have you considered 
 including cprof into Presta as a git submodule?
 
 -Ryan Dew
 
 On Jun 1, 2012, at 6:20 PM, Michael Blakeley wrote:
 
 https://github.com/mblakele/presta is now available. This library started 
 when I noticed the new 'result' option for xdmp:spawn in MarkLogic 5.0. 
 That looked handy, but xdmp:spawn still wanted me to supply a module path. 
 I wanted to write XQuery tasks with arbitrary code strings, like this:
 
  for $i in 1 to 4
  return xdmp:spawn-eval(
'xdmp:sleep(1000)', (),
options xmlns=xdmp:eval
  resulttrue/result
/options),
  xdmp:elapsed-time()
 
 ...and have all four tasks execute in parallel, using all my CPUs and 
 returning the results. Here's what I ended up with:
 
  import module namespace presta=com.blakeley.presta
at presta.xqy;
 
  let $presta-id := presta:prepare('xdmp:sleep(1000)')
  for $i in 1 to 4
  return presta:spawn(
$presta-id, (),
options xmlns=xdmp:eval
  resulttrue/result
/options),
  xdmp:elapsed-time()
 
 The presta:spawn function works just like xdmp:spawn, and takes the same 
 arguments - except that the first parameter is a presta id not a module 
 path. We get that id by calling presta:prepare, which takes the XQuery 
 string and stores it the Modules database.
 
 The id for a Presta module is based on xdmp:hash64, so preparing the same 
 XQuery twice will return the same id. Presta tries not to update the 
 Modules database any more than necessary - but it's still best to stash 
 these ids somewhere if you can.
 
 Once prepared, we can also use a presta id with presta:invoke. This is 
 handy for apps that eval the same strings frequently,  because the Presta 
 module works with the module cache. And we don't have to keep the XQuery 
 around after it has been registered. This could be a way for developers 
 using Java to get some of the performance benefits of invoke, without 
 having to write every XQuery module in advance.
 
 But there's more to MarkLogic than XQuery. We can also call presta:prepare 
 with XSLT, and then call presta:xslt-invoke.
 
 We need libraries, too. So you can call presta:import to install a library 
 module where other Presta functions can find it.
 
  presta:import(
'lib-test.xqy',
  'module namespace t=com.blakeley.presta.test;
   declare function t:now() { fn:current-dateTime() };')
 
  presta:invoke(
p:prepare(
  'import module namespace t=com.blakeley.presta.test
 at lib-test.xqy;
   t:now()
 
 I almost forgot about environments with multiple applications. Every 
 app-server automatically gets its own Presta environment, based on the 
 appserver id from xdmp:server(). If you want multiple servers to share the 
 same Presta modules, we can call presta:appkey-set('my-app-key'). If you 
 want to keep modules somewhere other than the Modules database, set the 
 app-server modules location to any database you like.
 
 Of course I also wanted Presta to work with profiling. Presta supports 
 cprof, and includes a copy of the cprof library. My plan is to discontinue 
 cprof as a separate project, and include it as part of Presta.
 
 Finally, Presta works with the security model. Presta installs a new role 
 that you can grant to any user.
 
 The code is at https://github.com/mblakele/presta - I hope it's useful.
 
 -- Mike

Re: [MarkLogic Dev General] HTML forms do not post into XQuery

[Ryan Dew]

I'm guessing it is because you are missing the name attributes on you're
inputs. You might want to try the following form:

form action=/cluster-add.xqy method=get class=form-horizontal
 fieldset style=border: 0px;
   legendAdd cluster/legend
   div class=control-group
 label class=control-label for=nameName/label
 div class=controls
   input class=input-xlarge id=name name=name type=text/
 /div
   /div
   div class=control-group
 label class=control-label for=idIdentifier/label
 div class=controls
   input class=input-xlarge id=id name=id
  style=font-family:monospace; type=text/
 /div
   /div
   input type=hidden id=aqt name=apt value=1/
   div class=form-actions
 button type=submit class=btn btn-primaryCreate/button
 button type=reset class=btnReset/button
   /div
 /fieldset
/form

Also, a best practice in web development is to not make updates on a GET
request. It is best to use POST for creating, PUT for updating, and DELETE
for deleting (Although I believe you have to make use of the
javascript XMLHttpRequest
or some other method for PUT and  DELETE since most, if not all, browsers
don't currently support PUT and DELETE at the moment.) You can use
xdmp:get-request-method() to check which method the request is using.

-Ryan Dew

On Tue, May 29, 2012 at 5:56 AM, Jashank Jeremy jash...@rulingia.comwrote:

 G'day,

 I'm still relatively new working with XQuery, so if I've done something
 blatantly and obviously wrong, please don't hurt me!

 I'm building a webapp in pure XQuery, and using it to teach myself the
 basics of XQuery.

 I have a '/cluster-add.xqy', which looks at the value of a hidden value
 (I need a better way to do it, but that's not critical) and switches
 between a form and an xdmp:document-insert().  If the form has
 method=get, nothing happens when I submit -- I just wind up back at an
 unpopulated '/cluster-add.xqy'.  If I go to
 '/cluster-add.xqy?aqt=1name=Testid=test', then the document-insert()
 runs.

 My program, when stripped down to the bare essentials, looks like this.

 return
 if (xdmp:get-request-field('aqt'))
 then
  (: inject content into the database :)
 else
  (: generate form :)

 The form itself looks like this.  I use Twitter Bootstrap to do styling.

 form action=/cluster-add.xqy method=get class=form-horizontal
  fieldset style=border: 0px;
legendAdd cluster/legend
div class=control-group
  label class=control-label for=nameName/label
  div class=controls
input class=input-xlarge id=name type=text/
  /div
/div
div class=control-group
  label class=control-label for=idIdentifier/label
  div class=controls
input class=input-xlarge id=id
   style=font-family:monospace; type=text/
  /div
/div
input type=hidden id=aqt value=1/
div class=form-actions
  button type=submit class=btn btn-primaryCreate/button
  button type=reset class=btnReset/button
/div
  /fieldset
 /form

 Any thoughts?

 Thanks in advance,

 Jashank

 --
 Jashank Jeremy
 PGP: 0x25A5C309


 ___
 General mailing list
 General@developer.marklogic.com
 http://community.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://community.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] getting MarkLogic to execute functions in parallel

[Ryan Dew]

I'm surprised that the for loop doesn't run xdmp:http-post in parallel. If
you really need the looping and parallel posting capability you could do
something like this:

let $endpoints := element endpoints {
element endpoint {http://localhost:8469/},
element endpoint {http://localhost:8469/},
element endpoint {http://localhost:8469/},
element endpoint {http://localhost:8469/},
element endpoint {http://localhost:8469/}
  }
let $range := 1 to count($endpoints/endpoint)
return
xdmp:eval(concat(
  declare variable $endpoints as element(endpoints) external;
  let ,string-join(for $i in $range
return concat($post,$i, :=
xdmp:http-post($endpoints/endpoint[,$i,])),','),
 return (, string-join(for $i in $range return
concat($post,$i),','),))
,(fn:QName(,endpoints),$endpoints))

Normally I would shy away from using xdmp:eval, but in this case the code
is tight enough that it doesn't open holes for XQuery injection (unless I'm
missing something, feel free to point it out) and you get the added
performance of running the posts in parallel, which can make a big
difference.

-Ryan Dew

On Wed, May 23, 2012 at 10:58 AM, seme...@hotmail.com
seme...@hotmail.comwrote:

  I have a search function that calls other search functions that are
 independent of each other. The master search function takes the results of
 these sub-search functions and compiles and processes them according to
 some rules.

 I would like the sub-search functions to all execute in parallel at the
 same time to reduce the response time of the call to the master search
 function.

 I tried using xdmp:http-post and xdmp:invoke both in for loops and also
 without for loops.

 for example:

 *xdmp:http-post (for loop) does not execute in parallel:

 let $sub-search-results :=
 for $sub-search in $sub-searches
 return xdmp:http-post($endpoint)

 let $subsearch-results :=
 subsearch-results
 {$sub-search-results}
 /subsearch-results

 return $sub-search-results


 *xdmp:http-post (no for loop) DOES execute in parallel:

 let $sub-search-result-1 := xdmp:http-post($endpoint)
 let $sub-search-result-2 := xdmp:http-post($endpoint)
 let $sub-search-result-3 := xdmp:http-post($endpoint)
 let $sub-search-result-4 := xdmp:http-post($endpoint)

 let $subsearch-results :=
 subsearch-results
 {$sub-search-result-1}
 {$sub-search-result-2}
 {$sub-search-result-3}
 {$sub-search-result-4}
 /subsearch-results

 return $sub-search-results


 I tried similar approaches using xdmp:invoke (both with and without a for
 loop) but neither would execute in parallel.

 Does this sound right? I was expecting to be able to get parallel
 execution in a for loop using xdmp:http-post (I even tried fn:unordered but
 it had not effect). and I was expecting to get parallel execution using
 xdmp:invoke somehow but I couldn't.

 The http-post approach isn't too bad for my situation but it feels like a
 little bit of a hack just to get parallel execution. Are there any other
 ways?

 thanks,
 Ryan

 ___
 General mailing list
 General@developer.marklogic.com
 http://community.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://community.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Is there a problem with xdmp:unpath

[Ryan Dew]

The second example doesn't provide a proper input. xdmp:unpath only handles
predicates dealing with position, for example  xdmp:unpath($x/doc/item/
itemid[2]).  In order to do something like what you want to you would need
to use xdmp:value, but be careful as that could open holes for XQuery
injection if user input is being used for the string you are passing to
xdmp:value.

-Ryan Dew

On Fri, May 4, 2012 at 12:13 PM, Darin McBeath ddmcbe...@yahoo.com wrote:

 If my  'unpath'  that contains a predicate, I get an error.

 For example,

 This will work,

 xdmp:unpath($x/doc/item/itemid)

 but this will not:

 xdmp:unpath($x/doc/item/itemid[@idtype='VAL'])

 I looked at the bug history (and fixed bugs) and didn't notice anything
 related to unpath.

 Curious if anyone else has noticed this, whether I'm doing something
 wrong, is there a workaround, etc.

 Thanks.

 Darin.

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

[MarkLogic Dev General] New Module for Memory Operations on XML

[Ryan Dew]

I've been working on my own module for updating XML in memory. It has
greater functionality than the module shipped with MarkLogic, such as
performing multiple operations at one time, and better performance from
what I have been able to measure. You can see my post on it at
http://maxdewpoint.blogspot.com/2012/04/lessons-learned-from-xquery-xml-memory.html.
I would love to get some input from the MarkLogic community on this.

-Ryan Dew
___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] New Module for Memory Operations on XML

[Ryan Dew]

The code itself can be found here: https://gist.github.com/2253100. So far
the operations available are replace, delete, insert-before, insert-after,
insert-child, insert-child-first. XQUF is a great resource for other
operations to add. Thanks for pointing that out I didn't think to look
there.

On Tue, Apr 17, 2012 at 8:02 AM, Geert Josten geert.jos...@dayon.nl wrote:

 Where can we find the code itself?



 And how much does it resemble the kind of updates allowed in XQUF?



 By the way, was kind of hoping MarkLogic would allow applying the xdmp
 node update functions (or copies of those) to in memory structures as well.
 Direct manipulation of the tree, without copying it recursively would be
 way faster..



 Kind regards,

 Geert





 *Van:* general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] *Namens *Ryan Dew
 *Verzonden:* dinsdag 17 april 2012 15:47
 *Aan:* MarkLogic Developer Discussion
 *Onderwerp:* [MarkLogic Dev General] New Module for Memory Operations on
 XML



 I've been working on my own module for updating XML in memory. It has
 greater functionality than the module shipped with MarkLogic, such as
 performing multiple operations at one time, and better performance from
 what I have been able to measure. You can see my post on it at
 http://maxdewpoint.blogspot.com/2012/04/lessons-learned-from-xquery-xml-memory.html.
 I would love to get some input from the MarkLogic community on this.



 -Ryan Dew

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] New Module for Memory Operations on XML

[Ryan Dew]

The function mapping idea is good. I'm not quite sure how I would
cts:highlight, I'll have to think on that one. I wanted to make it easy for
the module to be fully XQuery 1.0 compatible. Currently I have a commented
out code to replace the functionality of fn:generate-id (an XQuery 3.0
function) to generate a unique id for a node (mine is a little slower, but
the module still provides overall better performance). I might consider
forking it so one version is fully XQuery 1.0 compliant and another is
tailored to MarkLogic.

Thanks for the suggestions!

-Ryan Dew

On Tue, Apr 17, 2012 at 11:03 AM, Michael Blakeley m...@blakeley.comwrote:

 Geert, I expect that the xdmp update functions also operate by walking the
 input tree and copying it an output tree. Otherwise how would you have
 multi-version concurrency?

 But the xdmp functions are implemented in C++, which makes a difference.
 You might be able to quantify that difference by comparing
 xmdp:node-replace with the equivalent in-memory operations plus
 xdmp:document-insert. That kind of evidence could help persuade someone at
 MarkLogic that the feature would be worthwhile.

 Ryan, I think you could improve performance even more with judicious use
 of function mapping. It is often faster than FLWOR expressions are. You
 might also see if there is a way to use cts:highlight for some operations,
 since that is a C++ function.

 -- Mike

 On 17 Apr 2012, at 07:02 , Geert Josten wrote:

  Where can we find the code itself?
 
  And how much does it resemble the kind of updates allowed in XQUF?
 
  By the way, was kind of hoping MarkLogic would allow applying the xdmp
 node update functions (or copies of those) to in memory structures as well.
 Direct manipulation of the tree, without copying it recursively would be
 way faster..
 
  Kind regards,
  Geert
 
 
  Van: general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] Namens Ryan Dew
  Verzonden: dinsdag 17 april 2012 15:47
  Aan: MarkLogic Developer Discussion
  Onderwerp: [MarkLogic Dev General] New Module for Memory Operations on
 XML
 
  I've been working on my own module for updating XML in memory. It has
 greater functionality than the module shipped with MarkLogic, such as
 performing multiple operations at one time, and better performance from
 what I have been able to measure. You can see my post on it at
 http://maxdewpoint.blogspot.com/2012/04/lessons-learned-from-xquery-xml-memory.html.
 I would love to get some input from the MarkLogic community on this.
 
  -Ryan Dew
  ___
  General mailing list
  General@developer.marklogic.com
  http://developer.marklogic.com/mailman/listinfo/general

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] New Module for Memory Operations on XML

[Ryan Dew]

Here is a repo with some unit tests:
https://github.com/ryanjdew/XQuery-XML-Memory-Operations. Thanks to xray it
was really easy to do.

Thanks,
Ryan

On Tue, Apr 17, 2012 at 2:11 PM, Whitby, Rob, Springer Healthcare UK 
rob.whi...@springer.com wrote:

 **

 Hi Ryan,

 This is really interesting, thanks for sharing it.

 I recently encountered really poor performance using the in-mem-update
 module, and modified it slightly to use fn:generate-id().
 https://github.com/robwhitby/commons/tree/master/memupdate

 In my simple test of deleting nodes, the in-mem-update module takes 13.8s,
 modifying it to use fn:generate-id() improves this to 0.25s. I just tried
 your module and got 0.04s! Obviously this is just one use case but it's
 really impressive nonetheless. Do you have unit tests you could share on
 github? Or perhaps there are existing tests for in-mem-update that could be
 applied?

 Thanks again,
 Rob




 -Original Message-
 From: general-boun...@developer.marklogic.com on behalf of Ryan Dew
 Sent: Tue 4/17/2012 18:17
 To: MarkLogic Developer Discussion
 Subject: Re: [MarkLogic Dev General] New Module for Memory Operations on
 XML

 The function mapping idea is good. I'm not quite sure how I would
 cts:highlight, I'll have to think on that one. I wanted to make it easy for
 the module to be fully XQuery 1.0 compatible. Currently I have a commented
 out code to replace the functionality of fn:generate-id (an XQuery 3.0
 function) to generate a unique id for a node (mine is a little slower, but
 the module still provides overall better performance). I might consider
 forking it so one version is fully XQuery 1.0 compliant and another is
 tailored to MarkLogic.

 Thanks for the suggestions!

 -Ryan Dew

 On Tue, Apr 17, 2012 at 11:03 AM, Michael Blakeley m...@blakeley.com
 wrote:

  Geert, I expect that the xdmp update functions also operate by walking
 the
  input tree and copying it an output tree. Otherwise how would you have
  multi-version concurrency?
 
  But the xdmp functions are implemented in C++, which makes a difference.
  You might be able to quantify that difference by comparing
  xmdp:node-replace with the equivalent in-memory operations plus
  xdmp:document-insert. That kind of evidence could help persuade someone
 at
  MarkLogic that the feature would be worthwhile.
 
  Ryan, I think you could improve performance even more with judicious use
  of function mapping. It is often faster than FLWOR expressions are. You
  might also see if there is a way to use cts:highlight for some
 operations,
  since that is a C++ function.
 
  -- Mike
 
  On 17 Apr 2012, at 07:02 , Geert Josten wrote:
 
   Where can we find the code itself?
  
   And how much does it resemble the kind of updates allowed in XQUF?
  
   By the way, was kind of hoping MarkLogic would allow applying the xdmp
  node update functions (or copies of those) to in memory structures as
 well.
  Direct manipulation of the tree, without copying it recursively would be
  way faster..
  
   Kind regards,
   Geert
  
  
   Van: general-boun...@developer.marklogic.com [mailto:
  general-boun...@developer.marklogic.com] Namens Ryan Dew
   Verzonden: dinsdag 17 april 2012 15:47
   Aan: MarkLogic Developer Discussion
   Onderwerp: [MarkLogic Dev General] New Module for Memory Operations on
  XML
  
   I've been working on my own module for updating XML in memory. It has
  greater functionality than the module shipped with MarkLogic, such as
  performing multiple operations at one time, and better performance from
  what I have been able to measure. You can see my post on it at
 
 http://maxdewpoint.blogspot.com/2012/04/lessons-learned-from-xquery-xml-memory.html
 .
  I would love to get some input from the MarkLogic community on this.
  
   -Ryan Dew
   ___
   General mailing list
   General@developer.marklogic.com
   http://developer.marklogic.com/mailman/listinfo/general
 
  ___
  General mailing list
  General@developer.marklogic.com
  http://developer.marklogic.com/mailman/listinfo/general
 


 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Problem in making use of cts:not-query()

[Ryan Dew]

Not sure this will fix everything for you, but you might want to change
cts:element-value-query(xs:QName(id),*)  to
cts:element-value-query(xs:QName(id),*,
unwildcarded) or  cts:element-value-query(xs:QName(id),*, exact) since
* is a wildcard character.

On Wed, Mar 28, 2012 at 8:16 AM, Sudheer Yalaverthi 
sudheer.yalaver...@gmail.com wrote:

 I am using search:search. I need to make an additional query satisfying
 this condition.

 I can not make use of xpath here.


 On Wed, Mar 28, 2012 at 1:53 AM, VISH RAJPUT svishnu.sin...@gmail.comwrote:

 Hi Sudheer,

 You can use xpath expression with predicate like

 *//entrylist/entry[identity/id/text()='*' and
 not(permissionList/permission='read')]*

  to test the below highlighted.

 Regards,
 Vishnu


 On Wed, Mar 28, 2012 at 12:19 AM, Sudheer Yalaverthi 
 sudheer.yalaver...@gmail.com wrote:

 Hi Damon,

 I have made the xml simple.
 entrylist

 entry type=deny
 identity
 id*/id
 /identity
 permissionList
 permissionread/permission
 permissionwrite/permission
 /permissionList
 /entry
 entry type=deny
 identity
 id123/id

 /identity
 permissionList
 permissionwrite/permission
 /permissionList
 /entry
 /entrylist

 There will be 'n' number of xmls like the above in the DB. I need to
 retrieve the xml under two conditions.
 1. id = 123 and entry should have read permission in the permission list
  2. id = * and entry should not have the read permission in it

 I am able to form a query for the first condition. But i could not make
 the second condition.


 Thanks,
 Sudheer








 On Sun, Mar 25, 2012 at 9:09 AM, Damon Feldman 
 damon.feld...@marklogic.com wrote:

 Sudheer,

 ** **

 Can you post a self-contained, minimal example that runs in
 queryConsole? By this I mean a document insert or two and a query that
 shows the problem. Ideally, you would simplify the query and document to
 the simplest example that shows the behavior you are seeing, perhaps a
 single permission and two notes will work.

 ** **

 This allows people to paste it into queryConsole and immediately see
 your problem, then try solutions.

 ** **

 Yours,

 Damon

 ** **

 *From:* general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] *On Behalf Of *Sudheer
 Yalaverthi
 *Sent:* Friday, March 23, 2012 4:23 PM
 *To:* MarkLogic Developer Discussion
 *Subject:* Re: [MarkLogic Dev General] Problem in making use of
 cts:not-query()

 ** **

 John,

 i have tried this. But still this is not helping me.
 I am putting my exact xml , scenario.

 Xml in DB :

 notelist
 note
 textthis is my first note/text
 accessControl
 explicitAccessControlList
 entry type=deny
 identity
 id*/id
 /identity
 permissionList
 permissionread/permission
 permissionwrite/permission
 permissiondelete/permission
 permissionwrite-acl/permission
 /permissionList
 /entry
 entry type=allow
 identity
 id/id
 /identity
 permissionList
 permissionread/permission
 permissionwrite/permission
 permissiondelete/permission
 permissionwrite-acl/permission
 /permissionList
 /entry
 /explicitAccessControlList
 /accessControl
 /note
 note
 note
 textthis is my second note/text
 accessControl
 explicitAccessControlList
 entry type=deny
 identity
 id*/id
 /identity
 permissionList
 permissionwrite/permission
 permissiondelete/permission
 /permissionList
 /entry
 entry type=allow
 identity
 id/id
 /identity
 permissionList
 permissionread/permission
 permissionwrite/permission
 permissiondelete/permission
 /permissionList
 /entry
 /explicitAccessControlList
 /accessControl
 /note
 /notelist


 There will be so many xmls like this in the DB. My requirement is i
 have to retrieve this xml if i have a note element which has my qstring in
 the text, and has an entry(type=allow with 

Re: [MarkLogic Dev General] Fwd: [1.0-ml] XDMP-EXPNTREECACHEFULL

[Ryan Dew]

You could try a recursive function like the following. No guarantee it is
100% right, if you have sub elements that have the same names as your root
elements.

xquery version 1.0-ml;

declare function local:find-unique-qnames($found-qnames as xs:QName*) {
  let $next-qname := cts:search(collection()/*,
if (exists($found-qnames))
then cts:not-query(cts:element-query($found-qnames,cts:and-query((
else cts:and-query(())
  )[1]/node-name(.)
  return if (exists($next-qname))
  then local:find-unique-qnames(($found-qnames,$next-qname))
  else $found-qnames
};

declare function local:find-unique-qnames() {
  for $qn in local:find-unique-qnames(())
  order by string($qn)
  return $qn
};

local:find-unique-qnames()

On Mon, Mar 26, 2012 at 6:36 AM, Geert Josten geert.jos...@dayon.nl wrote:

 Hi Vishnu,



 It would help if you could explain why you need that list. But in general
 the best option would be to pre-calculate the list I guess. You can save it
 as a server-field (xdmp:set-server-field), to keep the list in memory on
 each host. But you would need an algorithm to initialize it, and each doc
 commit would have to check and update that list. The latter can be done
 with a post-commit trigger. The first can be done best by the strategy I
 already mentioned: divide all docs in chunks of 100 to 1000 docs, calculate
 distinct names of each chunk, and merge that somehow to the final list.



 You could also raise the tree size setting temporarily to do that initial
 calculation..



 Kind regards,

 Geert



 *Van:* general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] *Namens *VISH RAJPUT
 *Verzonden:* maandag 26 maart 2012 14:29

 *Aan:* MarkLogic Developer Discussion
 *Onderwerp:* Re: [MarkLogic Dev General] Fwd: [1.0-ml]
 XDMP-EXPNTREECACHEFULL



 Thanks Geert,



 Is there any alternate solution to find the unique elements within a
 database?



 Warm Regards,

 Vishnu





 On Mon, Mar 26, 2012 at 5:55 PM, Geert Josten geert.jos...@dayon.nl
 wrote:

 Hi Vishnu,



 90 mb isn’t much indeed, but MarkLogic is configured to keep a low memory
 footprint, even if there are 30 concurrent requests. To make that sure, the
 tree size limit (look at the database setting in the admin interface) is
 usually pretty low. I have 8Gb and still it is set to no more than 85mb by
 default. But you can increase it if you like.



 A more streaming approach like my advice attempts to achieve to some
 extend helps keeping the footprint low, and keep MarkLogic fast.



 Kind regards,

 Geert



 *Van:* general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] *Namens *VISH RAJPUT
 *Verzonden:* maandag 26 maart 2012 14:17
 *Aan:* MarkLogic Developer Discussion
 *Onderwerp:* Re: [MarkLogic Dev General] Fwd: [1.0-ml]
 XDMP-EXPNTREECACHEFULL



 Thanks Geert,



 But still it shows *XDMP-EXPNTREECACHEFULL: 
 distinct-values(collection(ContentAnalysis)//*/local-name()) --
 Expanded tree cache full on host *the database overall size is only
 90MB i don't think it is so huge data for marklogic





 Regards,

 Vishnu



 On Mon, Mar 26, 2012 at 1:25 PM, Geert Josten geert.jos...@dayon.nl
 wrote:

 Hi Vishnu,



 Your FLWOR expression won’t return distinct names, since you are applying
 the function to each individual name. You should write:



 distinct-values(

 for $a in //*

 return $a

 )



 Or better:



 distinct-values(collection()//*/local-name())



 But this still might not perform well, or still max out on list or tree
 caches. This approach is creating a complete list of all element names
 first, and starts applying distinct-values only thereafter. You might
 consider taking multiple steps, like per doc first, and then clustering per
 100 files, and only then all clusters. You could also just take 100 random
 samples, and use that. That doesn’t guarantee a 100% complete list, but it
 remains performant even if your database grows 10 or 100 fold.



 Kind regards,

 Geert



 *Van:* general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] *Namens *VISH RAJPUT
 *Verzonden:* maandag 26 maart 2012 8:29
 *Aan:* general@developer.marklogic.com
 *Onderwerp:* [MarkLogic Dev General] Fwd: [1.0-ml] XDMP-EXPNTREECACHEFULL



 The size of the all files is 90 MB approx.

 -- Forwarded message --
 From: *VISH RAJPUT* svishnu.sin...@gmail.com
 Date: Mon, Mar 26, 2012 at 11:56 AM
 Subject: [1.0-ml] XDMP-EXPNTREECACHEFULL
 To: general@developer.marklogic.com


 Hi,



 I have 2000 files in Marklogic database within a single forest and i want
 to find out the unique element name from this database for the whole 2000
 files. For this i wrote the below query:-



 for $a in //*

 return distinct-values($a/local-name()))



 but by this i got an error *[1.0-ml] XDMP-EXPNTREECACHEFULL * what
 should i do?





 Regards,

 Vishnu Singh




 

Re: [MarkLogic Dev General] phrase search containing stemmed results

[Ryan Dew]

That is how it is supposed to behave. You can pass the following search
options to change that behavior:

 options xmlns=http://marklogic.com/appservices/search;
term
term-optionunstemmed/term-option
/term
/options

On Thu, Mar 22, 2012 at 1:56 PM, Danny Sinang d.sin...@gmail.com wrote:


 I did a search:search('sneak out') but got results containing sneak
 out and sneaking out. Is this the supposed behavior ? Or is it a bug ?

 Is there a way to force search to return just exact match results without
 disabling stemmed search globally ?

 Regards,
 Danny

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] phrase search containing stemmed results

[Ryan Dew]

It this case it sounds like you want to surround the query with quotes,
like so: search:search('sneak out'). That is how you search for an exact
phrase.

On Thu, Mar 22, 2012 at 2:10 PM, Danny Sinang d.sin...@gmail.com wrote:

 Thanks Ryan.

 We have a central search module using the same search options.

 Is there a way to tell ML to do unstemmed searching just for phrase
 searches ?

 Regards,
 Danny


 On Thu, Mar 22, 2012 at 4:04 PM, Ryan Dew ryan.j@gmail.com wrote:

 That is how it is supposed to behave. You can pass the following search
 options to change that behavior:

  options xmlns=http://marklogic.com/appservices/search;
 term
 term-optionunstemmed/term-option
 /term
 /options

 On Thu, Mar 22, 2012 at 1:56 PM, Danny Sinang d.sin...@gmail.com wrote:


 I did a search:search('sneak out') but got results containing sneak
 out and sneaking out. Is this the supposed behavior ? Or is it a bug ?

 Is there a way to force search to return just exact match results
 without disabling stemmed search globally ?

 Regards,
 Danny

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general



 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Find exact query submitted

[Ryan Dew]

You can use cts:stem (http://api.xqueryhacker.com/#cts:stem ).

cts:stem('ran', 'en')
=
'run'

And you might want to look at cts:highlight.

http://api.xqueryhacker.com/#cts:highlight
Returns a copy of the node, replacing any text matching the query with the
specified expression. You can use this function to easily highlight any
text found in a query. Unlike fn:replace and other XQuery string functions
that match literal text, cts:highlight matches every term that matches the
search, including stemmed matches or matches with different capitalization.

On Wed, Mar 21, 2012 at 7:47 AM, Shah, Mehul (LNG-NPV) 
mehul.s...@lexisnexis.com wrote:

  I guess I should have asked question a bit differently with original
 purpose;

 ** **

 What we are looking for is hit highlighting in non-xml/text documents in
 its native form (like pdf, word). 

 One way we were thinking is to use expanded terms for a query and
 highlight in native document ourselves; 

 So we are looking for an option to return all possible term expansions for
 a query from marklogic - is it possible?

 ** **

 Is there any other way to hit-highlight in native document?



 ** **

 FYI:

 Previously I tried this:

 ...options xmlns=http://marklogic.com/appservices/search;

 term

   term-optionstemmed/term-option

 /term

   return-querytrue/return-query

 /options

 return search:search(ran,$options)

 ** **

 ** **

 I am expecting all possible expanded values it may be searching - e.g.
 ran, run, running...

 However I get following:

 ...search:qtextran/search:qtext

 -

 search:query

 -

 cts:word-query qtextref=cts:text

 cts:textran/cts:text

 cts:optionstemmed/cts:option

 /cts:word-query

 /search:query

 -

 search:metrics

 ...

 ** **

 ** **

 Thanks.

 ** **

 ** **

 *From:* general-boun...@developer.marklogic.com [mailto:
 general-boun...@developer.marklogic.com] *On Behalf Of *Ryan Dew
 *Sent:* Tuesday, March 20, 2012 6:45 PM
 *To:* MarkLogic Developer Discussion
 *Subject:* Re: [MarkLogic Dev General] Find exact query submitted

 ** **

 If this is for search:search you might want to add the following to your
 search options

 ** **

 return-querytrue/return-query

 -Ryan Dew

 On Tue, Mar 20, 2012 at 4:37 PM, Shah, Mehul (LNG-NPV) 
 mehul.s...@lexisnexis.com wrote:

 Is there a way to see exactly how the query was expanded and submitted
 finally?
 i.e. certain terms may be eliminated, certain terms may be stemmed, or
 there may be use of synonyms -- I want to know finally what was the query
 that was searched.

 Can I retrieve that as part of search result?

 Thanks.

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

 ** **

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Find exact query submitted

[Ryan Dew]

If this is for search:search you might want to add the following to your
search options

return-querytrue/return-query

-Ryan Dew

On Tue, Mar 20, 2012 at 4:37 PM, Shah, Mehul (LNG-NPV) 
mehul.s...@lexisnexis.com wrote:

 Is there a way to see exactly how the query was expanded and submitted
 finally?
 i.e. certain terms may be eliminated, certain terms may be stemmed, or
 there may be use of synonyms -- I want to know finally what was the query
 that was searched.

 Can I retrieve that as part of search result?

 Thanks.

 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] search:search - escaping quotes

[Ryan Dew]

With the default setting you shouldn't have to worry about escaping single
quotes. Search is flexible in that with the search:options element you can
pass options that modify the grammar. Unfortunately/fortunately, depending
of your view of things, the parser doesn't accommodate the use of regular
expression patterns in the grammar, so I don't think there is a simple
solution.

Your best bet may be to do your own parsing to create a cts:query and use
search:resolve. This isn't an ideal solution. When I have time I'd like to
look into it further.

-Ryan Dew


On Tue, Mar 20, 2012 at 9:03 AM, Danny Sinang d.sin...@gmail.com wrote:

 Hi,

 In search:search(), how do I escape single and double quotes entered by
 the user ?

 Regards,
 Danny



 ___
 General mailing list
 General@developer.marklogic.com
 http://developer.marklogic.com/mailman/listinfo/general


___
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general