The simplest is to have the document URI correspond to the element
value, and if you can use a random value it's good for concurrency.
If you can't do that, but you want to ensure only one document can have
a particular value for an element, I think it's pretty easy using
xdmp:lock-for-update() on an URI that corresponds to the element value.
You don't actually need to create a document at that URI, just use it to
serialize transactions. Here's one way to do it.
declare function lock-element-value($qn as xs:QName, $v as item)
{
xdmp:lock-for-update(
"http://acme.com/";
|| xdmp:hash64(fn:namespace-uri-from-QName($qn))
|| "/"
|| xdmp:hash64(fn:localname-from-QName($qn)))
};
You'd then do something like the following.
let $lock := lock-element-value($qn, $v)
let $existing := cts:search(fn:collection(), cts:element-range-query($qn, "=", $v,
"unfiltered"))
return
if (fn:exists($existing))
then ... do whatever you need to do with the existing document
else ... create a new document, safe from a race with another transaction
You'd want to use lock-element-value() in any updates that could affect
a change in the element value (insert, update, delete). I think you
could get away with ignoring deletes since those would automatically
serialize with any transaction that would modify the existing document.
We use this sort of pattern internally to ensure uniqueness of IDs.
Wayne.
On 06/04/2014 12:49 PM, Whitby, Rob wrote:
I thought 2 simultaneous transactions would both get read locks on the uri,
then one would get a write lock and the other would fail and retry. Maybe I'm
missing something though.
But anyway, I agree unique indexes would be a handy feature. e.g. our docs have
a DOI element which *should* be unique but occasionally aren't, would be nice
to enforce that rather than have to code defensively.
Rob
________________________________________
From: general-boun...@developer.marklogic.com
[general-boun...@developer.marklogic.com] on behalf of Ron Hitchens
[r...@ronsoft.com]
Sent: 04 June 2014 19:31
To: MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] New Feature Request: Unique Value Range
Indexes
Rob,
I believe there is a race condition here. A document may not exit as-of
the timestamp when this request starts running, but some other request could
create one while it's running. This request would then over-write that
document.
I'm actually more concerned about element values inside documents than
generating unique document URIs. It's easy to generate document URIs with
64-bit random numbers that are very unlikely to collide. But I want to
guarantee that some meaningful value inside a document is unique across all
documents.
In my case, the naming space is actually quite small because I want the IDs to be
meaningful but unique. For example "images:cats:fluffy:XX.png", where XX can
increment or be set randomly until the ID is unique. One way to check for uniqueness is
to make the document URI from this ID, then test for an existing document.
But this doesn't solve the general problem. I could conceivably have
multiple elements in the document that I want to be unique. To check for
unique element values it's necessary to run a cts query against the element(s).
And I'm not sure if you can completely close the race window between checking
for an existing instance and inserting a new one if the query comes back empty.
Someone from ML pointed out privately that checking for uniqueness in the
index would require cross-cluster communication. I'm sure that's true, but I'm
also pretty sure that any user-level code solution is going to be far less
efficient. I'd be happy to pay that ingestion time penalty for the guarantee
that indexed element values are unique. At query time, such a unique value
index should perform like any other range index.
---
Ron Hitchens {r...@overstory.co.uk} +44 7879 358212
On Jun 4, 2014, at 6:59 PM, "Whitby, Rob" <rob.whi...@springer.com> wrote:
How about something like this?
declare function unique-uri() {
let $uri := "/doc/" || xdmp:random() || ".xml"
return if (fn:not(fn:doc-available($uri))) then $uri else unique-uri()
};
I guess because indexes are distributed across forests, ensuring uniqueness is
not that easy?
Rob
________________________________________
From: general-boun...@developer.marklogic.com
[general-boun...@developer.marklogic.com] on behalf of Ron Hitchens
[r...@ronsoft.com]
Sent: 04 June 2014 18:01
To: MarkLogic Developer Discussion
Subject: [MarkLogic Dev General] New Feature Request: Unique Value Range
Indexes
I'm working on a project, one aspect of which requires minting unique IDs
and assuring that no two documents with the same ID wind up in the database. I
know how to accomplish this using locks (I'm pretty sure) but any such
implementation is awkward and prone to subtle edge case errors, and can be
difficult to test.
It seems to me that this is something that MarkLogic could do much more
reliably and quickly than any user-level code. The thought that occurred to me
is a variation on range indexes which only allow a single instance of any given
value.
Conventional range indexes work by creating term lists that look like this
(see Jason Hunter's ML Architecture paper), where each term list contains an
element (or attribute) value and a list of fragment IDs where that term exists.
aardvark | 23, 135, 469, 611
ant | 23, 469, 558, 611, 750
baboon | 53, 97, 469, 621
etc...
By making a range index like this but which only allows a single fragment ID
in the list, that would ensure that no two documents in the database contain a
given element with the same value. That is, attempting to add a second
document with the same element or attribute value would cause an exception.
And being a range index, it would provide a fast lexicon of all the current
unique values in the DB.
Such an index would look something like this:
abc3vk34 | 17
bkx46lkd | 52
bz1d34nm | 37
etc...
Usage could be something like this:
declare function create-new-id-doc ($id-root as xs:string) as xs:string
{
try {
let $id := $id-root || "-" || mylib:random-string(8)
let $uri := "/idregistry/id-" || $id
let $_ :=
xdmp:document-insert ($uri,
<registered-id>
<id>{ $id }</id>
<created>{ fn:current-dateTime() }</created>
</registered-id>
return $id
} catch (e) {
create-new-id-doc ($id-root)
}
};
This doesn't require that I write any (possibly buggy) mutual exclusion code
and I can be confident that once the xdmp:document-insert succeeds that the ID
is unique in the database and that the type (as configured for the range index)
is correct.
Any love for Unique Value Range Indexes in the next version of MarkLogic?
---
Ron Hitchens {r...@overstory.co.uk} +44 7879 358212
_______________________________________________
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general
--
Wayne Feick
Principal Engineer
MarkLogic Corporation
wayne.fe...@marklogic.com
Phone: +1 650 655 2378
www.marklogic.com
This e-mail and any accompanying attachments are confidential. The information
is intended solely for the use of the individual to whom it is addressed. Any
review, disclosure, copying, distribution, or use of this e-mail communication
by others is strictly prohibited. If you are not the intended recipient, please
notify us immediately by returning this message to the sender and delete all
copies. Thank you for your cooperation.
_______________________________________________
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general
