On 05/06/14 15:19, Andy Seaborne wrote:
(the query in question may just be syp[tom and not the root cause - that
is, it's not expensive but something else at the same time caue it to
go wrong)
That's what I thought too.
At a quick look:
?s text:query ('ampumavä*' 100000000)
Lucene is also taking RAM, partly the Lucene engine, partly to deal with
limit/unique results.
(and if Solr then it has to completely buffer responses as well).
It is Lucene, not Solr.
How many results are there to "?s text:query ("ampumavä*" 100000000 )"?
In fact zero, with that graph restriction (and I'm using the
graph-specific text index that I implemented). The large number is just
so that results would not be cut off by the text index for some edge
cases. So while this is a rather complex query, it should give an empty
result set fairly quickly.
It's too late now to check what size the journal was when this happened
yesterday, but I checked it for today, and all the three *jrnl files had
a size of zero and a timestamp of 04:57, which was the time when the
last updates finished. So at least last night, the flush completed very
soon after the update.
OK - it's highly unlikely to be updates then.
Good to know.
Though looking at how Fuseki behaves when I give it data, it seems that
the journal file size goes down to zero as soon as the update query
(using LOAD) completes, but there is still quite a lot of CPU usage for
at least 10 seconds after that even without any further queries. So
something is still happening after the journal has been emptied.
We have had to make sure bots didn't trigger expensive queries - for
example sorts. And had bots that ignore "no follow" (and robots.txt).
Right. I'm not worried about bot traffic, they don't normally generate
any more expensive queries than normal users.
We also use max=3 on the httpd front end to limit the impact of parallel
requests. Not perfect by any means but it does no harm.
Good idea, I could see if I can implement this kind of restriction in
Varnish which is in front of Fuseki.
-Osma
--
Osma Suominen
D.Sc. (Tech), Information Systems Specialist
National Library of Finland
P.O. Box 26 (Teollisuuskatu 23)
00014 HELSINGIN YLIOPISTO
Tel. +358 50 3199529
osma.suomi...@helsinki.fi
http://www.nationallibrary.fi
