Ah !!! ( see light bulb flashing over my head ) This is really really good, thank you ! I had seen examples searching on doc() and never really got the point. Your example finally unveils this, thank you ! Until now I've been wrapping searches and complex xpaths into user functions - which works but even within those functions I'd often have 10+ line expressions which were very hard to produce and nearly impossible to read. But by moving the search to doc() and putting *all* the expressions into the second argument lets me compose them ! (assign reasonable names to sub-expressions and even farm out complex parts to sub-functions).
I still would like to have it both ways :) -- but this is an approach that solves most of my issues reasonably elegantly. Thanks ! (now off to rewrite a lot of code ;) ... ---------------------------------------- David A. Lee Senior Principal Software Engineer Epocrates, Inc. d...@epocrates.com 812-482-5224 -----Original Message----- From: general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] On Behalf Of Damon Feldman Sent: Tuesday, July 19, 2011 9:12 AM To: General MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Async behavior of HTTP request David, You have some good points there. FWIW, I find that *semantically* most of these work as an optimizer, even though the actual implementation uses sub-expressions in a special way. In your case, I think you want to be able to refactor and compose code at will, so as a practical approach I suggest moving all query logic to the actual query portion, and just search on doc() as the searchable expression. This approach has no constraints on syntax. E.g. cts:search(xdmp:directory($mydir)//node, ()), can be rewritten as the fully-composable: let $node-q := cts:element-query(xs:QName("node"), cts:and-query(()) let $dir-and-node-q := cts:and-query((cts:directory-query($mydir), $node-q)) return cts:search(doc(), $dir-and-node-q) This form is fully composable, and you can use functions and variables anywhere you like. I hope this will work for you and make MarkLogic's approach seem more sensible. The syntactic sugar of embedded XPath and xdmp:directory() expressions are very convenient, and are still there if you want them, but this way you get full expressive power in a way that lets you break out any sub-expression into a variable, which is what I think you were describing earlier. Yours, Damon -----Original Message----- From: general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] On Behalf Of Lee, David Sent: Monday, July 18, 2011 10:43 PM To: General MarkLogic Developer Discussion Cc: General MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Async behavior of HTTP request I would trust an optimizer to never do a bad thing. Maybe not the best but at least never wrong. Otherwise its not an optimizer it's a bug factory. What I am most frustrated with is that I have to create long inline expressions to make them recognized as searchable. This goes against my 30 year experience of being able to break complex expressions into smaller easier to read and maintain expressions which in all other languages I've ever used are are actually optimized better by being broken up not worse. I'm happy to learn new things but this goes against every cs logic I have. To optimize ML queries I have to do the opposite of every standard cs technique I've ever learned... That is to refactor code to be huge single inline expressions at every chance ... Make it as long and ugly as possible in the hopes the runtime will find a searchable expression in static context and by every means AVOID breaking up long complex expressions into simpler sub expressions in blind hope the runtime discovers the meaning more efficiently. that hurts my purist brain, as my wife says in her wonderful southern phraseology .... its "just not right" I understand the technical reasons, but I don't like them. They are not a feature but rather a very heavy artificially implementation driven unnecessary limitation which is hard to program to in a manageable and scalable way. But the benefits are high so I put up with it. Being an experienced language developer I know that better can be done so I long for it. and expect it, and sad when it isnt delivered, I would be excited to be given the chance to implement it even if just to prove to myself how it can and should be done. Or perhaps be proven wrong and then I will eat some crow sandwich. In either case the challenge would be worth it That's my opinion. Sent from my iPad (excuse the terseness) David A Lee d...@calldei.com On Jul 18, 2011, at 8:02 PM, "Michael Blakeley" <m...@blakeley.com> wrote: > Special forms are a good way to think of it (and cts:highlight and cts:walk > are also special). Personally I've grown to appreciate having control over > when expressions are sent to the d-hosts in a cluster. Would you trust an > optimizer to do the right thing in every instance? > > -- Mike > > On 18 Jul 2011, at 15:11 , Evan Lenz wrote: > >> cts:search() and xdmp:estimate() are not functions per se, since their first >> argument must be an "expression" (and a searchable one at that). For >> cts:search(), the expression you provide is combined with the query you >> provide (in the second argument) to get the results from the index. Unlike a >> normal function, you can't pass just any evaluated node sequence as the >> first "argument." So they look like functions but aren't really. :-) >> (They're more like what are called "special forms" in Lisp.) >> >> Elsewhere, such expressions are evaluated normally, directly returning their >> results, which means that xdmp:directory() returns all the documents in the >> given directory. I agree that optimization of let clauses would be good, >> whether through lazy evaluation or some other approach. >> >> Evan Lenz >> Software Developer, Community >> http://developer.marklogic.com >> >> From: Geert Josten <geert.jos...@daidalos.nl> >> Reply-To: General MarkLogic Developer Discussion >> <general@developer.marklogic.com> >> Date: Mon, 18 Jul 2011 08:25:05 -0700 >> To: General MarkLogic Developer Discussion <general@developer.marklogic.com> >> Subject: Re: [MarkLogic Dev General] Async behavior of HTTP request >> >> Hi David, >> >> Well, one of the ML experts (Kelly or Jason perhaps?) remarked that >> searchable expressions are treated rather differently. They are not >> evaluated before being passed into functions like cts:search. Instead, they >> are somehow passed in as-is, and evaluated directly against indexes. >> Variables require evaluation before being passed in, so that would null the >> benefit of being evaluated against indexes. >> >> It might still be possible to optimize this, but it is different from lazy >> evaluation.. >> >> Kind regards, >> Geert >> >> Van: general-boun...@developer.marklogic.com >> [mailto:general-boun...@developer.marklogic.com] Namens Lee, David >> Verzonden: maandag 18 juli 2011 17:15 >> Aan: General MarkLogic Developer Discussion >> Onderwerp: Re: [MarkLogic Dev General] Async behavior of HTTP request >> >> This is very interesting, and also a source of confusion for me. >> >> Apparently ( and please correct me if I'm wrong) ... the runtime can lazily >> reference some kinds of variables (e.g. the one returned by xdmp:http-get()) >> but not others (say xdmp:directory() ) ... >> >> Example, if I put an otherwise searchable expression into a variable it >> becomes non-searchable >> >> return xdmp:directory(...)//node (: searchable :) >> vs. >> >> let $files := xdmp:directory( ...) , >> return $files//node (: not searchable :) >> >> >> Yet xdmp:http-get results are differed until referenced. (sweet). >> >> >> I understand this is a subtlety of optimization implementations but it would >> be really really nice if it were consistent (in the good way :) That is, >> assigning a searchable expression to a variable shouldn't make it >> non-searchable. >> >> Waiting for this feature in N.M(+1) :) >> >> >> >> >> >> ---------------------------------------- >> David A. Lee >> Senior Principal Software Engineer >> Epocrates, Inc. >> d...@epocrates.com >> 812-482-5224 >> >> From: general-boun...@developer.marklogic.com >> [mailto:general-boun...@developer.marklogic.com] On Behalf Of Abhishek53 S >> Sent: Friday, July 15, 2011 1:28 AM >> To: General MarkLogic Developer Discussion >> Subject: Re: [MarkLogic Dev General] Async behavior of HTTP request >> >> >> Thanks Danny ! Now the concept is much clear to me . >> >> Abhishek Srivastav >> Tata Consultancy Services >> Cell:- +91-9883389968 >> Mailto: abhishek5...@tcs.com >> Website: http://www.tcs.com >> ____________________________________________ >> Experience certainty. IT Services >> Business Solutions >> Outsourcing >> ____________________________________________ >> >> From: >> Danny Sokolsky <danny.sokol...@marklogic.com> >> To: >> General MarkLogic Developer Discussion <general@developer.marklogic.com> >> Date: >> 07/14/2011 11:41 PM >> Subject: >> Re: [MarkLogic Dev General] Async behavior of HTTP request >> Sent by: >> general-boun...@developer.marklogic.com >> >> >> >> >> Hi Abhishek, >> >> So it turns out the xdmp:http-gets will begin to evaluate asyncrounously. >> The sleeps that I put in that query forced them to block. They key is if >> you need to use the response or not. So this query does not wait for the >> http-gets to complete before logging in the return clause: >> >> xquery version "1.0-ml"; >> xdmp:log("Req # 1"), xdmp:sleep(1000), >> let $x := xdmp:http-get("http://marklogic.com";) >> let $y := xdmp:http-get("http://marklogic.com";) >> return >> xdmp:log("Process completed") >> >> I guess the question for you is do you need the query to use the output of >> the http-get. If so, then you have to wait for it to complete in order to >> complete the x-action. >> >> -Danny >> >> From: general-boun...@developer.marklogic.com >> [mailto:general-boun...@developer.marklogic.com] On Behalf Of Abhishek53 S >> Sent: Thursday, July 14, 2011 12:45 AM >> To: General MarkLogic Developer Discussion >> Subject: Re: [MarkLogic Dev General] Async behavior of HTTP request >> >> >> Danny >> >> Thanks for reply . Referring the link >> (http://www.marklogicevents.com/file/presentations/Effective_XQuery_in_MarkLogic_Heitkamp_Koul.pdf) >> where on slide 36 it is mentioned that HTTP interfaces executes >> asynchronously. Let me know If I misunderstood. >> >> Can It be possible to run child some transaction asynchronously. I am >> thinking of one way to achieve it where the child module is written into >> separate xqy file and then use xdmp:spawn to invoke the module. In this >> case the parent transaction will not wait for the response from the spawned >> module. Let me know If I am wrong here. >> >> Any separate way to achieve the asynchronous child transaction? >> >> Thanks >> Abhishek Srivastav >> Tata Consultancy Services >> Cell:- +91-9883389968 >> Mailto: abhishek5...@tcs.com >> Website: http://www.tcs.com >> ____________________________________________ >> Experience certainty. IT Services >> Business Solutions >> Outsourcing >> ____________________________________________ >> From: >> Danny Sokolsky <danny.sokol...@marklogic.com> >> To: >> General MarkLogic Developer Discussion <general@developer.marklogic.com> >> Date: >> 07/14/2011 06:27 AM >> Subject: >> Re: [MarkLogic Dev General] Async behavior of HTTP request >> Sent by: >> general-boun...@developer.marklogic.com >> >> >> >> >> >> >> >> >> Hi Abhishek, >> >> I think the http-get requests are *synchronous*, not asynchronous; that is, >> they will wait until they get the results to move on to the next thing in >> the query. So I am not sure I understand your question. For example, if >> you run the following: >> >> let $x := (xdmp:http-get("http://marklogic.com";), xdmp:log("Req # 1"), >> xdmp:sleep(1000)) >> let $y := (xdmp:http-get("http://marklogic.com";), xdmp:log("Req # 1"), >> xdmp:sleep(1000)) >> return >> xdmp:log("Process completed") >> >> You will see 3 log messages in your log, and they will be about 1 second >> apart from each other. >> >> 2011-07-13 17:53:48.395 Info: danny: Req # 1 >> 2011-07-13 17:53:49.462 Info: danny: Req # 1 >> 2011-07-13 17:53:50.462 Info: danny: Process completed >> >> -Danny >> >> From: general-boun...@developer.marklogic.com >> [mailto:general-boun...@developer.marklogic.com] On Behalf Of Abhishek53 S >> Sent: Wednesday, July 13, 2011 12:36 PM >> To: General MarkLogic Developer Discussion >> Subject: [MarkLogic Dev General] Async behavior of HTTP request >> >> Hi Folks, >> >> As per my understanding of HTTP request through Marklogic HTTP APIs is that >> they are asynchronous by nature.Between the HTTP service requests the >> behavior is asynchronous. >> >> Is the calling module execute asynchronously with invoked HTTP requests ie >> sample query >> >> let $x := xdmp:http-get($uri1,()) >> let $y := xdmp:http-get($uri2,()) >> return >> xdmp:log("Process completed") >> Can the logging be done irrespective to wait for the response from both the >> requests? >> >> Your suggestions will be appreciated. >> Thanks in advance >> Abhishek Srivastav >> Tata Consultancy Services >> Cell:- +91-9883389968 >> Mailto: abhishek5...@tcs.com >> Website: http://www.tcs.com >> ____________________________________________ >> Experience certainty. IT Services >> Business Solutions >> Outsourcing >> ____________________________________________ >> =====-----=====-----===== >> Notice: The information contained in this e-mail >> message and/or attachments to it may contain >> confidential or privileged information. If you are >> not the intended recipient, any dissemination, use, >> review, distribution, printing or copying of the >> information contained in this e-mail message >> and/or attachments to it are strictly prohibited. If >> you have received this communication in error, >> please notify us by reply e-mail or telephone and >> immediately and permanently delete the message >> and any attachments. Thank you >> _______________________________________________ >> General mailing list >> General@developer.marklogic.com >> http://developer.marklogic.com/mailman/listinfo/general >> _______________________________________________ >> General mailing list >> General@developer.marklogic.com >> http://developer.marklogic.com/mailman/listinfo/general >> >> _______________________________________________ >> General mailing list >> General@developer.marklogic.com >> http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > General@developer.marklogic.com > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list General@developer.marklogic.com http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list General@developer.marklogic.com http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list General@developer.marklogic.com http://developer.marklogic.com/mailman/listinfo/general
