[MarkLogic Dev General] : FreeText searching - Take Two

2009-11-19 Thread Kelly Stirman
David - that's why I suggested that your links point to a module that issues requests to fetch the most relevant document instead of providing a direct link. No sense in doing the work unless absolutely necessary. :) Kelly Message: 1 Date: Thu, 19 Nov 2009 13:16:40 -0800 From: "Lee, David" Su

RE: [MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Lee, David
I can always wish. An attribute worth indexing, IMHO is "number of children elements" or even "contains any child element". -Original Message- From: general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] On Behalf Of Geert Josten Sent: Thursday, November

RE: [MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Glidden, Douglass A
My impression (no guarantees) is that the cts:not-query *should* work and be equivalent to the XPath expression if the following two conditions are true: - The searchable expression (/terminology/conceptDef) is the fragment root, and - The definingConcepts and concept elements are guaranteed to ap

RE: [MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Danny Sokolsky
To really tell if it will help, try adding an xdmp:query-trace to your query and then look at how many fragments are selected. I did the following experiment and it seems Geert's not-query of an element-query has some promise: xquery version "1.0-ml"; xdmp:document-insert("a.xml", ), xdmp:docum

RE: [MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Geert Josten
Hi David, > > My guess is that the non-existance of elements is not > > indexed. I would think there would be some way in ML to > > index that information but I cant find it. > > It is pretty difficult to put something in an index that isn't there.. > :-) > > > I read in

RE: [MarkLogic Dev General] RE: FreeText searching - Take Two

2009-11-19 Thread Geert Josten
Hi David, Am I understanding correctly if I say you need a way to quickly determine the capitalized terms that are worth highlighting? Sounds like you are looking for one of the functions that (to put it short) return the keys from indexes: cts:words, cts:values, cts:element-words, etc.. You *

RE: [MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Glidden, Douglass A
A word of warning, unless you're 100% certain that your indices are set up such that the query inside the cts:not-query will _never_ need filtering, cts:not-query can miss correct results. The cts:not-query is one of the only queries that can err in failing to return positive matches (i.e. it g

RE: [MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Lee, David
Hi David, > Each conceptDef is a fragment (its parent, terminology is a > fragment Root) I rewrote it as this : I don't think that defining 'terminology' as fragment Root puts each conceptDef in its own fragment. Actually, as 'terminology' seems to be the root element of your document, it would

RE: [MarkLogic Dev General] RE: FreeText searching - Take Two

2009-11-19 Thread Lee, David
Thank you kelly. Yes I was missing $cts:queries But if I am reading this right, it doesnt fundamentally change the problem or solution. The problem being that I have to iteratively query for each potential matched term in order to generate a link for it. In this case the iteration is done inside

RE: [MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Geert Josten
Hi David, > Each conceptDef is a fragment (its parent, terminology is a > fragment Root) I rewrote it as this : I don't think that defining 'terminology' as fragment Root puts each conceptDef in its own fragment. Actually, as 'terminology' seems to be the root element of your document, it woul

[MarkLogic Dev General] RE: FreeText searching - Take Two

2009-11-19 Thread Kelly Stirman
David, This is what I thought you were trying to do: declare function local:new-text($query){ let $query-text := cts:word-query-text($query) let $link := fn:concat("default.xqy?search=",$query-text) let $count := xdmp:estimate(cts:search(collection(),$query)) return {$query-text} [{$c

RE: [MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Lee, David
Each conceptDef is a fragment (its parent, terminology is a fragment Root) I rewrote it as this : doc("/NDFRT/NDFRT_Public_2009.05.12_TDE.xml")/terminology/conceptDef[empty(definingConcepts/concept)] But it didnt speed things up much. My guess is that the non-existance of elements is not indexed

RE: [MarkLogic Dev General] Avoiding long running transactions

2009-11-19 Thread Runstein, Robert E. (Contr) (IS)
The 90 minute run occurred with 12GB memory available to MarkLogic. Increasing memory to 24GB reduced the run time to 13 minutes. Increasing to 31GB shortened the run to 12 minutes, so it looks like insufficient memory was the main cause. All the work is done in a single step within the pipeline,

Re: [MarkLogic Dev General] Avoiding long running transactions

2009-11-19 Thread Michael Blakeley
The approach you outline could help, or it might not help. I'd start by asking why the document processing is taking so much time. From the description, it isn't clear what part of document processing is the bottleneck. It sounds like you have a few good test cases: you might start by looking

[MarkLogic Dev General] RE: Avoiding long running transactions

2009-11-19 Thread Kelly Stirman
Bob Have you looked at RecordLoader? http://developer.marklogic.com/howto/tutorials/2006-06-recordloader.xqy Kelly Message: 2 Date: Thu, 19 Nov 2009 11:23:14 -0600 From: "Runstein, Robert E. (Contr) (IS)" Subject: [MarkLogic Dev General] Avoiding long running transactions To: Message-ID:

RE: [MarkLogic Dev General] How to run code when lock expires?

2009-11-19 Thread Danny Sokolsky
I see. No, there is no trigger event for updated or deleting a lock. When the lock is released, you can put some logic in the code that releases the lock to remove the shadow document at the same time as releasing the lock. That would not work for an expiring lock though. Maybe you could crea

[MarkLogic Dev General] Avoiding long running transactions

2009-11-19 Thread Runstein, Robert E. (Contr) (IS)
My application ingests documents that need to be broken up into subdocuments. We want the process to be atomic so our initial approach was to run in within a single CPF pipeline. While this works fine for small documents we have encountered larger documents that time out because processing take

RE: [MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Danny Sokolsky
The FLOWR is not likely to make it faster, as it has to do the same amount of work. I think you said that conceptDef is a fragment root, is that right? //conceptDef looks for all conceptDef elements at all levels of the tree. If you know the path to your conceptDef elements, that might help,

Re: [MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Andrew Welch
> doc("/NDFRT/NDFRT_Public_2009.05.12_TDE.xml")//conceptDef[empty(definingConcepts/concept)] > This is taking about 10 seconds right now. > Any suggestions ?   Thanks ! ...just a suggestion, but try rewriting it as a flowr expression, such as: for $c in //conceptDef where fn:empty($c/definingCo

[MarkLogic Dev General] [Announce] Poll for hands-on XML training offerings in Europe Q1'2010 (Mark Logic List)

2009-11-19 Thread G. Ken Holman
Please ignore the following if you have no interest in XML-related training. In Q1'2010 I will be in Europe a couple of times for some standards meetings (OASIS and ISO) and conferences ( http://www.XMLPrague.cz and PEPPOL) and I like to take advantage of my presence there by offering Crane's

[MarkLogic Dev General] search for non-existing elements

2009-11-19 Thread Lee, David
I'm learning that cts:search() is vastly faster then xpath expressions ... So I'm trying to come up with a cts:search() for the following doc("/NDFRT/NDFRT_Public_2009.05.12_TDE.xml")//conceptDef[empty(defining Concepts/concept)] This is searching for all conceptDef elements which have

[MarkLogic Dev General] RE: Character "Paragraph Separator"

2009-11-19 Thread Alberts, Alex (RB-NL)
Hi Danny, Answers 1) There are more documents with the same problem. I don't know how many. I have to write a query to check all the content. 2) That query doesn't throw the XDMP-URI exception. The result of this query is: /ReBIC/FBU/11/23/FBU_11_23_0823_solarfun-kader-1- De-erven-Brenninkmeije

[MarkLogic Dev General] File Upload Via HTML Form Problem

2009-11-19 Thread Neil Bradley
Hi, I am trying to use a form to upload an XML file into a MarkLogic database, but when my XQuery code tries to unquote the text in the package it reports that the file is incomplete (it is truncated). I have confirmed this. Here is the form I am using:

Re: [MarkLogic Dev General] Reg: Creating e-node and d-node withStandard Edition

2009-11-19 Thread dhivya vijayakumar
Hi Team, Do we need to have a separate “Failover license key” apart from the “Enterprise license key” to enable the forest level failover. We have Enterprise license key and created a cluster as well, but we don’t find the “Failover enable” option at groups page. Please advise us. with

Re: [MarkLogic Dev General] How to run code when lock expires?

2009-11-19 Thread Jeroen Pulles
Hi, When I take the lock, I also make a shadow copy of the document. I would like to remove that particular document when either the lock is released or the lock expires. regards, Jeroen On Wed, Nov 18, 2009 at 6:45 PM, Danny Sokolsky wrote: > Hi Jeroen, > > I am not quite sure what you are as