Re: [MarkLogic Dev General] xml ingestion quick question

2015-03-23 Thread Michael Blakeley
There is no nesting limit. Why can't you create an element-range index on attachment_filename? Is there an error message? What's the full version number of the MarkLogic release you're using? -- Mike On 23 Mar 2015, at 11:27 , cyanline llc i...@cyanline.com wrote: Hello, What am I

Re: [MarkLogic Dev General] Error while creating MarkLogic Forest with HDFS as data directory

2015-03-13 Thread Michael Blakeley
Are those curly braces in hdfs://{namdenode-hostname}:8020/tmp literal, or did you sanitize the log message? Is the spelling of namdenode-hostname correct? -- Mike On 13 Mar 2015, at 07:48 , Sudheer Yalaverthi sudheer.yalaver...@db.com wrote: 2015-03-12 19:17:20.087 Error: Automount

Re: [MarkLogic Dev General] Constructing JSON objects

2015-03-12 Thread Michael Blakeley
I tend to avoid declaring functions or variables in any namespace that I know other code is using. But you're doing it with your eyes open, and that's probably ok. Someday a server update may conflict. But if that happens, I'm sure you'll catch it in testing, and fix it before it breaks

Re: [MarkLogic Dev General] fn:replace Query

2015-03-11 Thread Michael Blakeley
You may want fn:translate instead of fn:replace. The docs for each function should make the difference clear. https://docs.marklogic.com/fn:translate https://docs.marklogic.com/fn:replace -- Mike On 11 Mar 2015, at 02:25 , Kapoor, Pragya pkapo...@innodata.com wrote: Hi, I am running

Re: [MarkLogic Dev General] Very puzzling bug in wildcard search results

2015-03-11 Thread Michael Blakeley
You might try 7.0-5, released on Friday. I believe at least one wildcard bug was fixed. Reindexing is a good idea too. Failing those, xdmp:plan or xdmp:query-trace might show some useful debugging info. -- Mike On 11 Mar 2015, at 11:22 , David Sewell dsew...@virginia.edu wrote: I'm trying

Re: [MarkLogic Dev General] Attribute seach in Marklogic

2015-03-11 Thread Michael Blakeley
If I understand correctly, you're asking how to match element-attribute-words without using an explicit query constraint? That is, you'd like q=alex to match an element-attribute like assessment/@title as well as ordinary word-query matches. If that's right,

Re: [MarkLogic Dev General] Constructing JSON objects

2015-03-11 Thread Michael Blakeley
I think Will is looking for a JSON syntax similar to map:new with map:entry args, which sometimes feels a little cleaner than calling map:map and map:put. As far as I know there's no direct equivalent for JSON. Until someone comes up with a better way, here are some utility functions that work

Re: [MarkLogic Dev General] General question on bi-temporal data

2015-03-11 Thread Michael Blakeley
Temporal documents are new to me too, so I hope I won't steer you off-course. What does your document-level data model look like? That is, what goes into each document when you call document-insert? I'd expect each document to model a single instrument and the price data for that instrument.

Re: [MarkLogic Dev General] 404 error on REST timestamp (v7.0-3)

2015-03-06 Thread Michael Blakeley
You're sure that the request is going to port 8001, not some other port? Does the response body have any additional information? What about ErrorLog.txt? The latest version is 7.0-4.3, and it's probably worth trying an upgrade: http://developer.marklogic.com/products/marklogic-server/7.0 — I

Re: [MarkLogic Dev General] roxy collection from existing db and app

2015-02-24 Thread Michael Blakeley
I suspect that both messages are related to properties that are referenced in the config XML but not defined. The first message looks like an empty appserver name. Check the config XML for http-server and xdbc-server elements. They should have an http-server-name or xdbc-server-name that won't

Re: [MarkLogic Dev General] What are the best practices for securing a Public ROXY REST Extension?

2015-02-24 Thread Michael Blakeley
The first rule for injection attacks is to avoid evaluating strings as code. This means avoiding xdmp:eval as well as xdmp:value and xdmp:unpath, although the latter function is a little less powerful and hence a little less dangerous. If you do need any eval-like function, use the weakest one

Re: [MarkLogic Dev General] Xray test framework 'not a node' error

2015-02-20 Thread Michael Blakeley
I see you opened https://github.com/robwhitby/xray/issues/42 — that's probably a better way to go, so I'll ask a few questions there. -- Mike On 20 Feb 2015, at 07:43 , Hasegawa, Margarete margarete.haseg...@ptsem.edu wrote: Hello, I've recently started working with the Xray test

Re: [MarkLogic Dev General] Preserving the HTML5 doctype

2015-02-18 Thread Michael Blakeley
MarkLogic doesn't store doctypes with XML documents. As I understand it this is mostly because they don't exist in the XQuery data model: http://www.w3.org/TR/xpath-datamodel/ Have you looked at Justin's post http://markmail.org/message/qmsos7np64ohyctp already? That approach presumes you can

Re: [MarkLogic Dev General] get developer key

2015-02-18 Thread Michael Blakeley
You should be able to register for a license key on any internet-connected host. Then enter it on the host you actually plan to use. Or I see this on the license key page: If you have purchased MarkLogic Server but do not know your license key, please send an email to lice...@marklogic.com. I

Re: [MarkLogic Dev General] query-meters(), and worse performance on a cluster

2015-02-17 Thread Michael Blakeley
The cause of the slowness is physics: you're asking the cluster to do quite a bit of work. The fix is to find a clever way to accomplish the same goal, whatever that goal may be. What's the larger goal that you're trying to accomplish? Got a user story? The qm:host data shows that you're

Re: [MarkLogic Dev General] Retry timeout?

2015-02-16 Thread Michael Blakeley
Try setting file-log-level=debug and then monitor the logs, to see if there is any evidence of retries. The observed performance problem may turn out to be unrelated to this retry timeout after all. But since it happens under load, deadlock detection is a pretty strong candidate. If so the

Re: [MarkLogic Dev General] How to set session value in ML

2015-02-16 Thread Michael Blakeley
Try passing values from those sessions into XQuery module variables, that are defined as external. See https://docs.marklogic.com/javadoc/xcc/overview-summary.html under Passing Variables With Queries. The examples use newAdhocQuery, but newModuleInvoke allows external variables too. Also

Re: [MarkLogic Dev General] Finding Top 10 words and collections or directories searching in java

2015-02-15 Thread Michael Blakeley
' is appearing the most with 2000 counts likewise 'inside' with 100 and so on , I need to show 1. neck (2000) 2. 'inside(100) and so on. It's long but some one can put me in the right direction. Thanks On Sat, Feb 14, 2015 at 1:20 AM, Michael Blakeley m...@blakeley.com wrote: Show us some XML

Re: [MarkLogic Dev General] Finding Top 10 words and collections or directories searching in java

2015-02-13 Thread Michael Blakeley
Show us some XML. It's difficult to decipher what you mean without concrete examples. Don't rule out anything at this point. You may need a new range index. You may have to use XQuery. -- Mike On 13 Feb 2015, at 10:50 , Maisnam Ns maisnam...@gmail.com wrote: Hi, Can someone help with

Re: [MarkLogic Dev General] Issues in PHP SDK for MarkLogic 8

2015-02-11 Thread Michael Blakeley
Try the REST API: https://docs.marklogic.com/REST/GET/v1/qbe — Mike On Feb 11, 2015, at 21:52, Srinivas srini...@nervecentrex.com wrote: Hi MarkLogicans, My Name Srinivas Guptha. I’ve been working on PHP, CakePHP(framework on PHP) since 2 years. Recently we came with a decision to work

Re: [MarkLogic Dev General] Access Log for Database clear

2015-02-11 Thread Michael Blakeley
(1) might be technically possible, but unlikely. More likely it was done through cq, which would not log anything special: just a POST to eval.xqy. Access logs don't usually include POST bodies, so you may not get anything conclusive there. However the ErrorLog may tell you exactly when the

Re: [MarkLogic Dev General] JSON number-node{} and boolean-node{} have no string-value

2015-02-09 Thread Michael Blakeley
This is still new stuff for me, but it seems like a bug. I'd expect string($node) and string(data($node)) to return the same result for any node with a simple type, because fn:string should atomize the node in much the same way fn:data does. In other words I'd expect it to behave pretty much

Re: [MarkLogic Dev General] Error 404 on getting developer license

2015-02-07 Thread Michael Blakeley
I think that's a bug at developer.marklogic.com — probably related to yesterday's release of MarkLogic 8. At the moment I don't have a workaround, but someone should be able to fix it fairly quickly. -- Mike On 7 Feb 2015, at 04:40 , Edgard Stéphane Konan sed.ko...@gmail.com wrote: Hello

Re: [MarkLogic Dev General] qconsole on a replica database

2015-02-06 Thread Michael Blakeley
That section of the docs says Replica databases can be queried but cannot be updated by applications. In my experience read-only queries are fine, but you'll see an error if you try any updates. Rob, where do you see that replica last updated message? In the admin UI replication status, or

Re: [MarkLogic Dev General] Content-Processing for docx Documents

2015-02-04 Thread Michael Blakeley
You didn't say if you'd checked error states: https://docs.marklogic.com/guide/cpf/error#id_13652 may be worth a try. Looking at your log it seems clear that the docx parts are being created, and I see no sign of errors. ... 2015-02-03 13:00:29.706 Info: [Event:id=CPF on-create]

Re: [MarkLogic Dev General] Content-Processing for docx Documents

2015-02-03 Thread Michael Blakeley
I would start by reading https://docs.marklogic.com/guide/cpf/error — especially https://docs.marklogic.com/guide/cpf/error#id_13652 - and also check the ErrorLog.txt for anything that looks CPF-related. -- Mike On 3 Feb 2015, at 05:18 , Mario Budi mario.budisc...@gmail.com wrote: Dear

Re: [MarkLogic Dev General] ERR_EMPTY_RESPONSE from security-install.xqy

2015-01-26 Thread Michael Blakeley
That segfault could be part of the problem. It may be that every time the server tries to bootstrap, it runs out of memory. According to https://developer.marklogic.com/products/marklogic-server/requirements-7.0 the minimum is still 512-MB RAM, but at least 2-GB is recommended. -- Mike On

Re: [MarkLogic Dev General] Can MLCP use an input query ?

2015-01-23 Thread Michael Blakeley
Under the covers it looks like mlcp evaluates the document_selector as part of a FLWOR. So it won't use the URI lexicon, and probably doesn't need to because of the way mlcp works. Anyway if the supplied document_selector uses the indexes properly then it would be no less efficient than what

Re: [MarkLogic Dev General] Data Rebalancing in Cluster Archtitecture: “Statistical” Assignment Policy

2015-01-21 Thread Michael Blakeley
I'm not Haitao, but I'll mention that I generally prefer bucket assignment to statistical assignment. I like bucket assignment because it's deterministic: given a URI, the server can compute which forest it should live in. That means the server knows which forest ought to contain a given

Re: [MarkLogic Dev General] Reading Documents

2015-01-20 Thread Michael Blakeley
...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] On Behalf Of Michael Blakeley Sent: Wednesday, January 21, 2015 12:20 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Reading Documents I'm not quite sure what you mean, but http://docs.marklogic.com

Re: [MarkLogic Dev General] Reading Documents

2015-01-20 Thread Michael Blakeley
I'm not quite sure what you mean, but http://docs.marklogic.com/cts:distinctive-terms might help. — Mike On Jan 20, 2015, at 22:20, Preethi Padmati preethi_padm...@infosys.com wrote: Hi All, Can you please let me know if I can read a complex text

Re: [MarkLogic Dev General] Searching Standalone Properties Documents

2015-01-19 Thread Michael Blakeley
Try `cts:search(xdmp:document-properties(), $query)`. Similar to fn:doc or fn:collection, the zero-arg form of xdmp:document-properties returns all properties. -- Mike On 19 Jan 2015, at 09:28 , neil bradley n...@bradley.co.uk wrote: Hi, I am having a problem working out how (if it is

Re: [MarkLogic Dev General] can a milestone tag be a fragment root?

2015-01-19 Thread Michael Blakeley
:02 , Craig A. Berry craigbe...@mac.com wrote: On Jan 18, 2015, at 9:14 PM, Michael Blakeley m...@blakeley.com wrote: Thanks for the reply. Adding fragment rules makes sense if and only if you have large documents with a number of elements that form conceptually equivalent sub-documents

Re: [MarkLogic Dev General] can a milestone tag be a fragment root?

2015-01-18 Thread Michael Blakeley
No, not in any useful way. But don't give up hope. Technically you could configure a fragment root on an empty element, but that would only hurt performance. Every empty element would create subfragment overhead, adding N extra empty child fragments to each document. But there wouldn't be any

Re: [MarkLogic Dev General] Bulk content processing in MarkLogic

2015-01-16 Thread Michael Blakeley
Taskbot has been available for almost a year now. As often happens I noticed I was writing repetitive code, and tried to abstract it. Hopefully it's useful for others too. Using $tb:OPTIONS-SYNC doesn't avoid timeouts. It just means that the caller will wait for any results, instead of

Re: [MarkLogic Dev General] A very general queston

2015-01-15 Thread Michael Blakeley
One way to think of this is that XPath is fully standard, and is like SQL. XQuery is a layer on top of that, like PL/SQL or T-SQL. The higher-level language is very capable, but when you use it you are likely to include some product-specific code too. You can isolate use of product-specific

Re: [MarkLogic Dev General] Proxy sever query

2015-01-13 Thread Michael Blakeley
Your code sample won't evaluate. The FLWOR lacks a return, which is required. Then the function declaration follows the body, which isn't allowed. Then the code declares local:http-get-proxy, but calls the undefined function xdmp:http-get-proxy. Try the code at

Re: [MarkLogic Dev General] Marklogic performance estimation

2015-01-13 Thread Michael Blakeley
The output from prof:eval will probably make your queries look slower than they really are, because profiling adds its own overhead to the evaluation. Profiling shows where queries spend time, but those timings will include profiler overhead. If you're just trying to get the server elapsed

Re: [MarkLogic Dev General] MarkLogic Concurrent Threads/latency

2015-01-12 Thread Michael Blakeley
If you need more threads, use more threads. Without knowing the details of the workload it's impossible to do more than guess about the right number. I generally start with thread count = 2*CPUs. If the workload is inherently high-latency and low-CPU then I might use 4x or even more. When

Re: [MarkLogic Dev General] MarkLogic Concurrent Threads/latency

2015-01-09 Thread Michael Blakeley
The server is keeping some idle threads active to handle incoming requests. That's meant to improve performance. A thread may also spend a small amount of time cleaning up after a request completes, but this should be minimal. If you're trying to track down a latency issue I think you should

Re: [MarkLogic Dev General] function index in MarkLogic?

2015-01-08 Thread Michael Blakeley
Specifying a collation at the app-server doesn't fully solve the problem. If a collation is the right tool for the job, it needs to be part of the index and again as part of the query. It's important to understand the difference between specifying a collation in a range index, in a FLWOR

Re: [MarkLogic Dev General] Code Coverage for Xquery

2014-12-23 Thread Michael Blakeley
There's https://github.com/robwhitby/xray/tree/v2.1 — still somewhat experimental, but it might help. — Mike On Dec 23, 2014, at 22:12, lakshmidevi.sainathsubraman...@cognizant.com lakshmidevi.sainathsubraman...@cognizant.com wrote: Hi, One of our client is having Web service

Re: [MarkLogic Dev General] Code Coverage for Xquery

2014-12-23 Thread Michael Blakeley
if there are any other options. Thanks, Lakshmi. From: general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] On Behalf Of Michael Blakeley Sent: Wednesday, December 24, 2014 12:04 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General

Re: [MarkLogic Dev General] XDMP-CONFLICTINGUPDATES

2014-12-21 Thread Michael Blakeley
Take a look at https://blakeley.com/blogofile/2013/06/21/introduction-to-multi-statement-transactions/ — it talks about using MST with DLS. -- Mike On Dec 21, 2014, at 22:17, Kapoor, Pragya pkapo...@innodata.com wrote: Hi, I need to run multistatement transcations in if else using dls

Re: [MarkLogic Dev General] What's the best Multitenancy approach for 5, 000 tenants in a shared MarkLogic Cluster?

2014-12-18 Thread Michael Blakeley
I don't think thousands of forests on one host will be practical. Just to pick one problem, each forest writes to its label once per second. So thousands of forests will drive thousands of writes/sec, without even running any queries. So I'd rethink the physical isolation: document permissions

Re: [MarkLogic Dev General] xray test framework and security problem

2014-12-17 Thread Michael Blakeley
The instructions may be a little out of date. But with 7.0-4 I needed these: xdmp:document-get xdmp:eval xdmp:filesystem-directory xdmp:filesystem-file-exists xdmp:invoke xdmp:xslt-invoke I didn't need xdmp:filesystem-file, at least not for the tests I ran. One debugging trick for security is

Re: [MarkLogic Dev General] MarkLogic use case

2014-12-15 Thread Michael Blakeley
Range indexes require XML. Take a look at https://docs.marklogic.com/guide/cpf/default for PDF. You'll have some work to do selecting the right content to feed into your range index. JSON should be easy too, because it has an XML representation. For csv and the others you are basically on

Re: [MarkLogic Dev General] ways to execute xQuery against

2014-12-15 Thread Michael Blakeley
Yes, using https://docs.marklogic.com/xdmp:get-request-field -- Mike On 15 Dec 2014, at 18:28 , Alexei Betin abe...@elevate.com wrote: Thanks, Danny Yes, this is pretty much what I am after, except is there also a way to pass parameters to my query? E.g., if I did curl

Re: [MarkLogic Dev General] Reducing number of forests

2014-12-13 Thread Michael Blakeley
Why do you expect reducing the number of forests to improve performance? Depending on circumstances it could, but more often I see too few forests. I usually size systems with one main content forest per two CPU cores. If you don't have enough content to fill out that many forests then reducing

Re: [MarkLogic Dev General] efficiently limit documents count by dateTime field

2014-12-11 Thread Michael Blakeley
You are on the right track: you'll want a dateTime element range index on SubmitDate. However I suspect the way the query is written is causing problems. You could check this using xdmp:plan. The problem may be that you're doing too much work in an XPath predicate. It's easy to pretend that an

Re: [MarkLogic Dev General] Is it possible to add range index from command line without bootstrap?

2014-12-09 Thread Michael Blakeley
Have you looked at https://docs.marklogic.com/admin:database-add-range-element-index yet? I'm not clear on what you mean by invalid documents. But if you create a range index and reindexing is enabled, then each forest will automatically reingest documents as needed. There's no need to delete

Re: [MarkLogic Dev General] Optimal buffer size when streaming out files

2014-12-05 Thread Michael Blakeley
4096 is pretty small as buffers go, but it's a decent size and it matches up well with some OS and filesystem structures. Larger buffers will only help if your documents tend to be larger than 4-kiB, and if nothing else gets in the way. For maximum effect you'll also want to tune the client OS

Re: [MarkLogic Dev General] Use XQSync to copy files to zip archive without metadata files

2014-11-20 Thread Michael Blakeley
Start by pasting in all the XQSync properties you're setting. Anonymize anything confidential: the database password for example. OUTPUT_FILTER_FORMATS only supports document formats: binary(), text(), xml. This is more or less copied from the XCC

Re: [MarkLogic Dev General] Namespaces are getting removed

2014-11-20 Thread Michael Blakeley
It can be dangerous to use fn:name in that context. Try fn:node-name instead. The fn:name function returns a string, which can do strange things depending on the namespace declarations in context. But fn:node-name returns a QName. Consider this code: declare namespace t=cat ; let $a := t:a/

Re: [MarkLogic Dev General] Path Range Queries Default Collation.

2014-11-14 Thread Michael Blakeley
https://docs.marklogic.com/guide/search-dev/encodings_collations describes the differences between the root collation and the codepoint collation. The root collation adds more useful case and diacritic sensitivity to the Unicode codepoint order, so it will make more sensible sort orders when

Re: [MarkLogic Dev General] XQuery Vs XSLT - Any Performance benefit

2014-11-11 Thread Michael Blakeley
That matches my experience: I can generally get some performance improvement by porting XSLT work to XQuery. However I'd start by choosing the right tool for the job, and getting the data model and processing model right. Then you can do any optimization work as you see performance problems. I

Re: [MarkLogic Dev General] Parameterizing SPARQL Queries

2014-11-11 Thread Michael Blakeley
Have you seen http://stackoverflow.com/questions/24661476/marklogic-how-to-pass-integer-to-semsparql-triples-in-bindings yet? -- Mike On 11 Nov 2014, at 15:07 , c-bl...@uchicago.edu blair.char...@gmail.com wrote: I can't seem to paramaterize LIMIT and OFFSET in a SPARQL query. This is

Re: [MarkLogic Dev General] Path-Range-index vs Element-Attribute-Range index

2014-11-06 Thread Michael Blakeley
If //foo/@bar is the right precision, use an element-attribute range index. If you need to specify a more general or a more precise XPath, use a path range index. Given the same precision there should be no difference in query performance, because either index would contain the same values.

Re: [MarkLogic Dev General] Executing CORB with a time limit

2014-11-04 Thread Michael Blakeley
Using the `timeout` command sounds simple and effective. Another approach: Write your corb task module to check for a server field and test its dateTime value against current-dateTime. If the job has timed out according to this check, throw an error. Before starting corb, arrange to set that

Re: [MarkLogic Dev General] User-triggered Unstemmed Full Text Searches

2014-10-30 Thread Michael Blakeley
The docs probably refer to filtering with cts:contains and a rewritten version of the query. I wouldn't do that, because it means reimplementing much of search:search and it won't provide accurate counts or facets. Instead if you can afford a little extra dis space I would enable both fast

Re: [MarkLogic Dev General] Help in xray code coverage

2014-10-27 Thread Michael Blakeley
Once there's an open issue in github I could take a look at this. At a minimum, it looks like the xqy path and module root need to be regex-escaped before being passed to any regex functions. Those unescaped backslashes in the windows paths are a problem. I also see

Re: [MarkLogic Dev General] Response times in access logs

2014-10-03 Thread Michael Blakeley
No, but it's often a good idea to run a transparent reverse proxy in front of MarkLogic: nginx, apache, or whatever you prefer. You can track response times there. Another solution is to instrument your XQuery by calling xdmp:log whenever xdmp:elapsed-time exceeds some threshold at the end of

Re: [MarkLogic Dev General] Point in time queries

2014-10-02 Thread Michael Blakeley
, at 07:33 , shruti kapoor shrutikapoor@gmail.com wrote: Thanks Mike. Could you tell me the exact use for point in time queries besides fores rollover. On 1 Oct 2014 20:48, Michael Blakeley m...@blakeley.com wrote: You don't, not using point-in-time queries. It sounds like you want

Re: [MarkLogic Dev General] order by clause ignored?

2014-10-02 Thread Michael Blakeley
The order of operations in a FLWOR isn't guaranteed. The return expression could be evaluated before the sorting is done. You might also be interested to know that maps are unordered. So you can put those items into the map in any order you like, but the keys will end up in whatever internal

Re: [MarkLogic Dev General] order by clause ignored?

2014-10-02 Thread Michael Blakeley
That is not necessary, because an untyped simple element will atomize as xs:untypedAtomic and then sort as a string. The string representation of an xs:dateTime orders exactly like an xs:dateTime. Try it. -- Mike On 2 Oct 2014, at 09:34 , Joe Bryan joe.br...@marklogic.com wrote: Hi Rachel,

Re: [MarkLogic Dev General] order by clause ignored?

2014-10-02 Thread Michael Blakeley
xquery provider, it's not just a quirk of marklogic optimisation. Thanks for the lessons! Rachel -Original Message- From: Michael Blakeley m...@blakeley.com Reply-To: MarkLogic Developer Discussion general@developer.marklogic.com Date: Thursday, 2 October 2014 17:42 To: MarkLogic

Re: [MarkLogic Dev General] phrase-arounds

2014-10-02 Thread Michael Blakeley
If I understand your use-case correctly, you want a word-query exclude: https://docs.marklogic.com/guide/admin/wordquery Phrase-around is for phrase control: https://docs.marklogic.com/guide/admin/text_index#id_29138 -- Mike On 2 Oct 2014, at 12:05 , Girish Kulkarni girishbkulka...@gmail.com

Re: [MarkLogic Dev General] Point in time queries

2014-10-01 Thread Michael Blakeley
You don't, not using point-in-time queries. It sounds like you want version control rather than point-in-time. Take a look at DLS: https://docs.marklogic.com/guide/app-dev/dls -- Mike On 1 Oct 2014, at 00:15 , Kapoor, Shruti skapo...@innodata.com wrote: Hi all, How do I retrieve all the

Re: [MarkLogic Dev General] Adding new index to replicated ML servers

2014-09-29 Thread Michael Blakeley
Change the replica first, then the master. Each will index independently. -- Mike On Sep 28, 2014, at 23:04, vi...@tilaton.fi wrote: I tried to search for the answer, but couldn't find it, I would appreciate if any of you could direct me to the correct chunk of documentation to find this

Re: [MarkLogic Dev General] Minutely scheduled job is executing more than once within a minute - reg.,

2014-09-15 Thread Michael Blakeley
Retrying updates is one way the server resolves deadlocks. You don't want to turn that off, and you can't. Instead you should rethink the way your scheduled job works. First I would gather more information by setting file-log-level=debug. Then let the scheduled task run and look for

Re: [MarkLogic Dev General] possible bug w/directory deletion

2014-08-28 Thread Michael Blakeley
With 7.0-3 on OSX and directory-creation=manual, I don't see a XDMP-DOCNOTFOUND error either. xdmp:describe(doc(/test/)); xdmp:describe(xdmp:document-properties(/test/)); xdmp:directory-create(/test/); xdmp:describe(xdmp:document-properties(/test/)); xdmp:document-delete(/test/);

Re: [MarkLogic Dev General] How to send HTML file attachment using xdmp:email

2014-08-28 Thread Michael Blakeley
Have you looked at the last example at https://docs.marklogic.com/xdmp:email yet? This example demonstrates sending a message with attachments. Looks to me like it would be simple to change the Content-Type. You'd probably have to use https://docs.marklogic.com/xdmp:base64-encode instead of

Re: [MarkLogic Dev General] transaction and query in distributed storage

2014-08-27 Thread Michael Blakeley
A database configuration can include any number of forests. The database guarantees the ACID properties of the transaction, no matter how many forests are configured. You can rely on those ACID properties. Joins are a different matter: more important for queries rather than updates. Joins are

Re: [MarkLogic Dev General] Issues when using xmlgraphics-commons-1.5

2014-08-22 Thread Michael Blakeley
What's the full version of MarkLogic? If you don't have the latest support release, try upgrading. Assuming that doesn't help... what's the server doing while the browser is spinning? You said you've already checked ErrorLog.txt? Have you tried setting file-log-level to debug or higher?

Re: [MarkLogic Dev General] Attribute indexing

2014-08-21 Thread Michael Blakeley
Use xdmp:plan to see why your query was slow. xdmp:plan(/transaction/*[@transInfoRef eq ti1]) qry:final-plan qry:and-query qry:term-query weight=0 qry:key5327439496900942259/qry:key qry:annotationdescendant(doc-root(element(transaction),doc-kind(document))

Re: [MarkLogic Dev General] 2 elements in a single document- satisfying/not satisfying And not query.

2014-08-21 Thread Michael Blakeley
Why use cts:and-not-query at all? Why use a word-query when you want to match values? You want documents with party1 attribute 'abc'. Ask for them. let $party1 := abc return cts:search( doc(), cts:element-attribute-value-query( xs:QName(shortname), xs:QName(party1), $party1))) This is

Re: [MarkLogic Dev General] xdmp:http-get wrapper function (encoding, ssl certs, redirects, ...)?

2014-08-20 Thread Michael Blakeley
As I understand it the main reason to check certificates is to avoid man-in-the-middle attacks. Do you care if a hostile party intercepts your request and returns its own response, instead of the actual site response? In the past when I've needed a really robust spider, I've used existing

Re: [MarkLogic Dev General] XQuery processing time

2014-08-19 Thread Michael Blakeley
The problem is likely '*'. Be specific about your QNames, because MarkLogic indexes element values and element-attribute values. Use xdmp:plan to see what indexes are used. To avoid an ugly XPath consider using cts:search with a cts:element-attribute-value query. That allows a sequence of

Re: [MarkLogic Dev General] Attribute indexing

2014-08-19 Thread Michael Blakeley
MarkLogic always indexes element values and element-attribute values in a hash index. No extra configuration is needed, and it can't be turned off. Element, attribute, and path range indexes are value indexes. These are only needed for fast sorting, inequality lookups, facets, and similar

Re: [MarkLogic Dev General] Java webservice/XCC security

2014-08-18 Thread Michael Blakeley
XCC authentication already does digest authentication with whatever username and password you supply. That's pretty secure, and SSL/TLS helps too. But it sounds like you want to treat those credentials as a soft target, and instead authorize each update as needed? If I understand the use-case

Re: [MarkLogic Dev General] Spawning tasks in clusters

2014-08-15 Thread Michael Blakeley
As you have discovered, you must call xdmp:spawn on the host where you want the task to run. In this respect the task server acts like any other app-server: configuration is shared by the entire group, but each instance runs independently. I agree that it would be handy to have a 'host' option

Re: [MarkLogic Dev General] Search Snippet

2014-08-09 Thread Michael Blakeley
That sounds like the `per-match-tokens` option for the built-in snippet function: https://docs.marklogic.com/guide/search-dev/query-options#id_36346 -- Mike On 9 Aug 2014, at 09:18 , Dinesh jdin...@xedge.com.sg wrote: Hi, Does anyone have an example of using custom code to customize snippet

Re: [MarkLogic Dev General] Recently added

2014-08-08 Thread Michael Blakeley
If there are no search terms then documents will appear in native database order, also called document order. This is something like RDBMS row order. Generally speaking that won't be the same as the insertion order. So how can we get the most recent documents in a reliable way? If you have

Re: [MarkLogic Dev General] cts:element-word-query

2014-08-05 Thread Michael Blakeley
You can use the string representation of a boolean with an element-word-query term. But an ewq for 'true' would also match 'fubar true' because it's a word match. It's a little better to use cts:element-value-query, but that's still just a lexical match. So 'true' won't match the valid

Re: [MarkLogic Dev General] Best way to normalize non-ASCII characters to ASCII?

2014-08-01 Thread Michael Blakeley
I don't know of a library or built-in that would handle that. But you could write one. If you do, try to release the source. In another direction, one fairly cheap solution might be to check the user-name before creating the user, or try-catch the library call. If it looks iffy or fails,

Re: [MarkLogic Dev General] XQuery difference betwee windows and linux platforms

2014-08-01 Thread Michael Blakeley
I must have run into the same problem at some point, because cq's lib-xquery has an to xs:unsignedLong constructor in similar code: xs:unsignedLong( ($dt - xs:dateTime('1970-01-01T00:00:00Z')) div xs:dayTimeDuration('PT1S')) I didn't remember the details, so I recreated the error with

Re: [MarkLogic Dev General] Transaction control and errors

2014-07-30 Thread Michael Blakeley
: general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] On Behalf Of Michael Blakeley Sent: 29 July 2014 19:17 To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Transaction control and errors I'd expect rollback to be automatic

Re: [MarkLogic Dev General] Security Design

2014-07-30 Thread Michael Blakeley
other users. Documents sizes will range between 100kb and 500kb. Thanks for your help. Cheers, Tim On Mon, Jul 28, 2014 at 7:50 PM, Michael Blakeley m...@blakeley.com wrote: In the builtin security model, simply adding a collection to a document won't make that document visible

Re: [MarkLogic Dev General] SPARQL Random Timeout

2014-07-30 Thread Michael Blakeley
The latest release is 7.0-3 and includes several bug fixes related to triples and SPARQL. So I'd try upgrading first, and see if that helps. If that doesn't help, it could be that it works when enough data is in cache, and times out when it has to do too much I/O because data isn't cached. You

Re: [MarkLogic Dev General] SPARQL Random Timeout

2014-07-30 Thread Michael Blakeley
, at about 2% of the total hits. One thing I noticed is that when I turn optimization off (optimize=0), the query runs fine. When optimize is set to 1 or 2, I start seeing the timeout issues again. Any thoughts on that? Thanks, Joel On Wed, Jul 30, 2014 at 12:10 PM, Michael Blakeley

Re: [MarkLogic Dev General] Facet Resolution Time Differentiation

2014-07-30 Thread Michael Blakeley
First I would try 7.0-3. If that doesn't help, the difference might be due to qconsole running as an admin user while the REST call isn't. Admin users ignore permissions, while non-admin users resolve them. That can make a difference in performance. Either way 2-6 seconds doesn't sound good.

Re: [MarkLogic Dev General] Transaction control and errors

2014-07-29 Thread Michael Blakeley
I'd expect rollback to be automatic if the transaction ends with an uncaught error. You should only have to worry about rollback and commit if the statement runs in MST update mode and completes successfully. To test that take a look at the host-status page in the admin UI. The third table

Re: [MarkLogic Dev General] XSLT check if result document has been created

2014-07-29 Thread Michael Blakeley
If you want to skip the duplicates, one common pattern is to track them using a map. You could also use distinct-values, but a map can be more efficient. let $m-seen := map:map() for $i in $list let $key := $i/@id/string() (: Or some other expr based on $i :) let $is-duplicate :=

Re: [MarkLogic Dev General] Forest not available, open replica

2014-07-29 Thread Michael Blakeley
I'd take that one to support. They'll want a full support log: those two log messages aren't much information to go on. You haven't told us which release you're running, but it's always worthwhile to check http://developer.marklogic.com/download for an upgrade. If there is one, it might fix

Re: [MarkLogic Dev General] Security Design

2014-07-28 Thread Michael Blakeley
In the builtin security model, simply adding a collection to a document won't make that document visible to another user. Visibility of documents is controlled by read permissions and roles. Granting read permission for a document to a role makes that document visible to users that have that

Re: [MarkLogic Dev General] mlcp export problem/question

2014-07-24 Thread Michael Blakeley
In my experience XQSync and RecordLoader often have problems due to GC pauses. Starting the JVM with `-Xincgc` helps. I don't think mlcp does that automatically, but from looking at mlcp.sh I think it will pick up the JVM_OPTS environment variable. So `export JVM_OPTS=-Xincgc` should work.

Re: [MarkLogic Dev General] Deadlock issue with xdmp:lock-for-update

2014-07-18 Thread Michael Blakeley
• What's the full version number? • On what OS? • How many hosts are in the cluster? • How many forests on each host? • Is automatic directory creation enabled? As written the description of the problem doesn't quite make sense to me. How can SVC-EXTIME unnoticed... lead to XDMP-DEADLOCK?

Re: [MarkLogic Dev General] MD5 - Hash Question

2014-07-18 Thread Michael Blakeley
If security is important, keep http://en.wikipedia.org/wiki/Md5#Security in mind and use a stronger hash function. MarkLogic also provides members of the http://en.wikipedia.org/wiki/SHA2 family. https://docs.marklogic.com/xdmp:sha256 https://docs.marklogic.com/xdmp:sha384

Re: [MarkLogic Dev General] Documentation/Examples for cts:count-aggregate

2014-07-18 Thread Michael Blakeley
The docs probably expect prior experience with the search guide and with other lexicon functions: https://docs.marklogic.com/guide/search-dev/aggregate https://docs.marklogic.com/cts:element-values https://docs.marklogic.com/cts:values Here's a simple example: for $c in $companies let

Re: [MarkLogic Dev General] XDMP-NOTXN: No transaction with identifier

2014-07-16 Thread Michael Blakeley
First I'd try the latest XCC jars. There's an effort to maintain forward compatibility, but the gap from 5 to 7 may be too much. http://developer.marklogic.com/products/xcc Is there a reason why you're setting DocumentFormat.NONE? If possible, I'd set it to the appropriate value: XML or BINARY

  1   2   3   4   5   6   7   8   9   10   >