Hi Osma.
Please don't apologise. It's nice to have someone to discuss this with
who is so far up the curve.
I was just going to rewrite exactly how you suggest (giving in?) but you
beat me to it.
SELECT ?score ?ent
WHERE {
?ent spatial:nearby(51.507999420166016 -0.10999999940395355
70.01807880401611 'km') .
{
{ ?ent rdfs:label ?label .
FILTER (regex(?label, "environment", "i") &&
langmatches(lang(?label), "en"))
}
UNION
{ ?ent rdfs:comment ?comment .
FILTER (regex(?comment, "environment", "i") &&
langmatches(lang(?comment), "en")) }
}
?ent rdf:type iotic:Entity .
}
gets the same results (sans ?score, but who cares?) and completes in
61ms! No contest.
Thanks for your help
Mark
Technology Lead, Iotic Labs
+44 7973 674404
mark.whar...@iotic-labs.com
https://www.iotic-labs.com
On 21/12/15 14:30, Osma Suominen wrote:
> Hi Mark!
>
> Thanks for trying my queries. I'm sorry (but not surprised!) to hear
> that they weren't any better than your original query.
>
> I think JENA-999 is really the key here - until it is implemented, I
> don't know any way of speeding this up. My (very fuzzy) understanding of
> ARQ is that it always performs joins with iterators, so there is no way
> to first retrieve both result sets (one from spatial, one from
> jena-text) and only then join them, unless you do the cross product and
> only limit with FILTER, which is even slower (options 2 and 3).
>
> By the way you didn't say which version of Jena you're using. Jena 3.0.1
> actually does include the initial JENA-999 patch which, despite its
> problems (related to retrieving score and especially the original
> literal - see JENA-1093), might actually work in your case. It was
> reverted only after the 3.0.1 release.
>
> Other than testing with 3.0.1, I'd suggest skipping jena-text for now
> and changing the text query part to a FILTER expression instead, with a
> REGEX or CONTAINS. Applying that kind of filter to 450 items shouldn't
> add much to the execution time of the spatial query.
>
> -Osma
>
> [1] https://issues.apache.org/jira/browse/JENA-999
> [2] https://issues.apache.org/jira/browse/JENA-1093
>
> On 21/12/15 13:02, Mark Wharton wrote:
>> Hi Osma.
>>
>> Thanks for your help. It was exactly the kind of help that I wanted.
>>
>> Haha, but life is rarely that simple...
>>
>> I ran all your versions and my original a couple of times each to see if
>> there was any difference in performance.
>>
>> query time1 time2
>> Orig 4.615 4.632
>> 1 4.648 4.608
>> 2 6.385 6.401
>> 3 6.353 6.442
>>
>> I think that proves that my "naive" version is working how I thought,
>> (and the same as your version 1). i.e. getting all the results from the
>> spatial predicate and then individually checking them against the text
>> predicate. (Which is why it's slower if you swap the predicates over,
>> as there are more matches on the text).
>>
>> This kind of query can't be that unusual, surely? Things with "foo" in
>> their text within 50Km of point "bar"?
>>
>> Mark
>>
>>
>> Technology Lead, Iotic Labs
>> +44 7973 674404
>> mark.whar...@iotic-labs.com
>> https://www.iotic-labs.com
>>
>> On 21/12/15 07:53, Osma Suominen wrote:
>>> Hi Mark!
>>>
>>> I'm not sure that the jena-external-index approach would help. It might
>>> or might not, depending on how it's implemented. AFAIK it's just an idea
>>> right now, I haven't seen any code.
>>>
>>> In any case I think the problem with jena-text and probably jena-spatial
>>> too (not very familiar with it) is that they are pretty fast when you
>>> can get by doing just a single query with an unbound subject. But when
>>> you instead have a fixed subject, or more likely a list of possible
>>> subjects, the performance will be very bad because every subject will be
>>> queried separately from the Lucene index. This is probably what happens
>>> with your query - the results of one query will be fed to the other.
>>> JENA-999 tried to address some of this by introducing a cache into
>>> jena-text, but the patch that was committed had other problems (it
>>> returned the wrong results in some cases) so it was reverted recently
>>> and an improved version hasn't yet been developed.
>>>
>>> In any case, you could try these variants of the original query:
>>> (I'm really on thin ground here, but I spent a couple of days last week
>>> trying to optimize a somewhat similar query involving jena-text and
>>> various other conditions, trying to find a solution with good
>>> performance)
>>>
>>> 1. Try to isolate the text and spatial queries with extra braces
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>> 70.01807880401611 'km') }
>>> { (?ent ?score) text:query ('environment' 'lang:en') }
>>> ?ent rdf:type iotic:Entity .
>>> }
>>>
>>> 2. Use a FILTER expression to delay matching the results
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>> 70.01807880401611 'km') .
>>> (?ent2 ?score) text:query ('environment' 'lang:en') .
>>> FILTER (?ent = ?ent2)
>>> ?ent rdf:type iotic:Entity .
>>> }
>>>
>>> 3. Combination of above ideas
>>>
>>> SELECT ?score ?ent
>>> WHERE {
>>> { ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>> 70.01807880401611 'km') }
>>> { (?ent2 ?score) text:query ('environment' 'lang:en') }
>>> FILTER (?ent = ?ent2)
>>> ?ent rdf:type iotic:Entity .
>>> }
>>>
>>>
>>> -Osma
>>>
>>>
>>> On 21/12/15 00:28, Mark Wharton wrote:
>>>> Hi Marco.
>>>>
>>>> Yes, that's it. The indexes work well in isolation, but don't combine
>>>> well. Smooshing them into a single index would be a great idea,
>>>> especially if the query could resolve both text and spatial predicates
>>>> with one matching scan of the index.
>>>>
>>>> Perhaps Stephen could be persuaded to pick up the pace on this one?
>>>>
>>>> Thanks Mark
>>>>
>>>> On 20 December 2015 12:53:39 GMT+00:00, Marco Neumann
>>>> <marco.neum...@gmail.com> wrote:
>>>>> yes correct Mark I am only referring to the extra payload here for
>>>>> invoking the spatial filter in the SPARQL query.
>>>>>
>>>>> now that you mention a particular issue with the combined use of both
>>>>> jena-text and jena-spatial (something I am not aware of ) this might
>>>>> be related to duplicated code in the two projects. back in May Stephen
>>>>> Allen wrote on the dev-list that he is about to address some of this
>>>>> possibly in a new jena-external-index project.
>>>>>
>>>>> http://mail-archives.apache.org/mod_mbox/jena-dev/201505.mbox/%3ccaptxtvpwu2ijogyj0kx8o6-07yokk5g1t32b_k3g_cjaqvk...@mail.gmail.com%3E
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Dec 20, 2015 at 2:29 AM, Mark Wharton
>>>>> <mark.whar...@iotic-labs.com> wrote:
>>>>>> Hi
>>>>>>
>>>>>> Thanks for this. I've read the chapter in the book and now I'm not
>>>>> sure
>>>>>> if I misunderstand your reply or you've only addressed half of the
>>>>> problem.
>>>>>>
>>>>>> I'm not worried about the performance of the spatial search in
>>>>> isolation
>>>>>> - that's 97ms which is fine. The text search on its own takes a bit
>>>>>> longer but that's acceptable, too.
>>>>>>
>>>>>> It's when I put the spatial and text *together* that query time
>>>>> increase
>>>>>> by 10-30 times. That's the bit I don't understand and would like
>>>>> some
>>>>>> help with.
>>>>>>
>>>>>> Is there a SPARQL query formulation that can "AND the indexes" rather
>>>>>> than retrieving one set and looping through to retrieve the matches
>>>>>> individually on the other. (Which is my guess as to how it works).
>>>>>>
>>>>>> Thanks for your help so far.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> Technology Lead, Iotic Labs
>>>>>> mark.whar...@iotic-labs.com
>>>>>> https://www.iotic-labs.com
>>>>>>
>>>>>> On 18/12/15 18:59, Marco Neumann wrote:
>>>>>>> it's a common spatial access method latency in paticular for small
>>>>>>> data sets. you can try a mbr range query instead.
>>>>>>>
>>>>>>> see Chapter 13 Managing Space and Time in Semantic Web Programming
>>>>> by
>>>>>>> John Hebeler et. al.. 2009
>>>>>>>
>>>>>>> On Fri, Dec 18, 2015 at 10:13 AM, Mark Wharton
>>>>>>> <mark.whar...@iotic-labs.com> wrote:
>>>>>>>> Hi Jena users.
>>>>>>>>
>>>>>>>> I'm having performance problems with a query that uses text and
>>>>> location
>>>>>>>> search
>>>>>>>>
>>>>>>>> The query is roughly this:
>>>>>>>>
>>>>>>>>
>>>>>>>> SELECT ?score ?ent
>>>>>>>> WHERE {
>>>>>>>> ?ent spatial:nearby(51.507999420166016 -0.10999999940395355
>>>>>>>> 70.01807880401611 'km') .
>>>>>>>> (?ent ?score) text:query ('environment' 'lang:en') .
>>>>>>>> ?ent rdf:type iotic:Entity .
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> There are about 450 entities in that radius
>>>>>>>> There are about 2200 entities with environment in their
>>>>> rdfs:comment
>>>>>>>>
>>>>>>>> The query takes 5 seconds.
>>>>>>>>
>>>>>>>> I've tried this:
>>>>>>>> Commenting out the text predicate the query takes 97 ms
>>>>>>>> Commenting out the spatial predicate the query takes 438 ms
>>>>>>>> Swapping the spatial and text predicates it takes 15 seconds
>>>>>>>>
>>>>>>>>
>>>>>>>> My question is this... It looks like the query is separately
>>>>> getting
>>>>>>>> the results of the first two predicates and merging (somehow) to
>>>>> find
>>>>>>>> the intersection. Is there a formulation which will intersect the
>>>>> two
>>>>>>>> sets faster?
>>>>>>>>
>>>>>>>> Many TIAs,
>>>>>>>>
>>>>>>>> Mark
>>>>>>>> --
>>>>>>>> Technology Lead, Iotic Labs
>>>>>>>> mark.whar...@iotic-labs.com
>>>>>>>> https://www.iotic-labs.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> ---
>>>>> Marco Neumann
>>>>> KONA
>>>>
>>>
>>>
>
>
