Hi Regis
Please see this thread where I experienced a similar issue:
https://issues.apache.org/jira/browse/JENA-181
A couple of possible solutions is either to insert delays between
queries (not possible in your scenario) or to set
-XX:MaxDirectMemorySize to a higher value than the default
Andy - is it possible that the Service code is not calling close() on
the query iterators in a timely fashion (i.e. keeping too many
connections open thus exhausting the direct memory buffer) as that
turned out to be the main culprit in my case from re-reading that thread.
Rob
On 4/9/12 11:36 AM, Regis Pires Magalhães wrote:
Andy,
Yes, I'm running the server and using the client with direct access to
Fuseki (i.e. without proxy).
When the Exception occurs in client side, the server outputs to log:
*...*
*14:23:22 INFO Fuseki :: [443] 200 OK*
*14:23:22 INFO Fuseki :: [444] GET
http://fuseki.mooo.com:8080/dbpedia-geo/sparql?query=SELECT++*%0AWHERE%0A++%7B+%3Chttp%3A%2F%2Fdbpedia.org%2Fresource%2FDonaldsonville%252C_Louisiana%3E+%3Chttp%3A%2F%2Fwww.w3.org%2F2003%2F01%2Fgeo%2Fwgs84_pos%23lat%3E+%3Flat+.%0A++++%3Chttp%3A%2F%2Fdbpedia.org%2Fresource%2FDonaldsonville%252C_Louisiana%3E+%3Chttp%3A%2F%2Fwww.w3.org%2F2003%2F01%2Fgeo%2Fwgs84_pos%23long%3E+%3Flong%0A++%7D%0A
*
*14:23:22 INFO Fuseki :: [444] Query = SELECT * WHERE {<
http://dbpedia.org/resource/Donaldsonville%2C_Louisiana> <
http://www.w3.org/2003/01/geo/wgs84_pos#lat> ?lat .<
http://dbpedia.org/resource/Donaldsonville%2C_Louisiana> <
http://www.w3.org/2003/01/geo/wgs84_pos#long> ?long } *
*14:23:22 INFO Fuseki :: [444] OK/select*
*14:23:22 WARN Fuseki :: [444] RC = 500 : Direct buffer
memory*
When I use SERVICE SILENT, the client outputs:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 15:34:19 WARN QueryIterService
:: SERVICE: HttpException: 500 Server Error
15:34:20 WARN QueryIterService :: SERVICE: HttpException: 500 Server
Error
15:34:21 WARN QueryIterService :: SERVICE: HttpException: 500 Server
Error
15:34:21 WARN QueryIterService :: SERVICE: HttpException: 500 Server
Error
15:34:22 WARN QueryIterService :: SERVICE: HttpException: 500 Server
Error
15:34:23 WARN QueryIterService :: SERVICE: HttpException: 500 Server
Error
15:34:23 WARN QueryIterService :: SERVICE: HttpException: 500 Server
Error
15:34:24 WARN QueryIterService :: SERVICE: HttpException: 500 Server
Error
15:34:25 WARN QueryIterService :: SERVICE: HttpException: 500 Server
Error
...
How can I avoid "Direct buffer memory" problem?
The client uses *jena-arq-2.9.0-incubating.jar* and the server uses *
Fuseki-0.2.1-incubating-SNAPSHOT*.
Regards,
Regis.
On Mon, Apr 9, 2012 at 1:17 PM, Andy Seaborne<[email protected]> wrote:
Regis,
What is fuseki.mooo.com? Are you running that server?
500 means the server is unhappy -- the server log file may give further
details.
I don't know why you are seeing large numbers of connections - the client
code uses vanilla java.net HTTP code so it should reuse connections or
close them (at least in the ARQ 2.9.0).
I'm not familiar with NginX as a proxy. ARQ does try to execute more
specific SERVICE requests that use one vague one - sometimes the right
thing to do, sometimes not. The engine can't tell.
We did recently (post 2.9.0 release - available in 2.9.1-SNAPSHOT) change
the SERVICE handling code. If you say "SERVICE SILENT" it will read all
the results from a call before moving on - that might help.
Andy
On 08/04/12 14:00, Regis Pires Magalhães wrote:
This code can be used to see the HttpException (500 Server Error) remotely
(only 18 results found before the exception occurs):
import com.hp.hpl.jena.query.**QueryExecution;
import com.hp.hpl.jena.query.**QueryExecutionFactory;
import com.hp.hpl.jena.query.**QueryFactory;
import com.hp.hpl.jena.query.**QuerySolution;
import com.hp.hpl.jena.query.**ResultSet;
import com.hp.hpl.jena.rdf.model.**ModelFactory;
import com.hp.hpl.jena.sparql.core.**DatasetImpl;
public class Query {
public static void main(String[] args) {
String query = "PREFIX owl:<http://www.w3.org/2002/**
07/owl#<http://www.w3.org/2002/07/owl#>>\n"
+
"PREFIX geopos:<
http://www.w3.org/2003/01/geo/**wgs84_pos#<http://www.w3.org/2003/01/geo/wgs84_pos#>>\n"
+
"select ?s ?lat ?long where {\n" +
" SERVICE
<http://*fuseki.mooo.com:8080***/lgd-dbpedia/sparql>
{\n" +
" select * where {\n" +
" ?s owl:sameAs ?geo .\n" +
" FILTER regex(str(?geo),\"dbpedia\")\
**n"
+
" } \n" +
" }\n" +
" SERVICE
<http://*fuseki.mooo.com:8080***/dbpedia-geo/sparql>
{\n" +
" select * where{\n" +
" ?geo geopos:lat ?lat ;\n" +
" geopos:long ?long .\n" +
" } \n" +
" } \n" +
"} \n";
long start = System.currentTimeMillis();
QueryExecution exec =
QueryExecutionFactory.create(**QueryFactory.create(query),
new
DatasetImpl(ModelFactory.**createDefaultModel()));
ResultSet rs = exec.execSelect();
int n = 0;
while (rs.hasNext()) {
rs.next();
System.out.print(++n + " ");
}
long time = System.currentTimeMillis() - start;
System.out.println(n + " results found in " + time +
"ms.");
}
}
Abraços,
Regis.
2012/4/8 Regis Pires
Magalhães<regispiresmag@gmail.**com<[email protected]>
Andy,
The exception is always the same NoRouteToHostException and we the only
proxy we use between the client and the Fuseki server is a reverse proxy
(NginX).
I've also observed that the number of network connections with the server
increases from 0 to a bit more than 28000, when the exception
occurs. Almost all of the connections are TIME_WAIT.
When I try a direct connection to Fuseki (even locally using 127.0.0.1 as
the address to the Sparql Endpoint) I only get 17 results (the number of
connections increases from 0 to 212) and the exception changes to:
Exception in thread "main" HttpException: HttpException: 500 Server Error
: HttpException: 500 Server Error
at
com.hp.hpl.jena.sparql.engine.**http.HttpQuery.execCommon(**
HttpQuery.java:375)
at com.hp.hpl.jena.sparql.engine.**http.HttpQuery.execGet(**
HttpQuery.java:202)
at com.hp.hpl.jena.sparql.engine.**http.HttpQuery.exec(HttpQuery.**
java:157)
at com.hp.hpl.jena.sparql.engine.**http.Service.exec(Service.**java:71)
at
com.hp.hpl.jena.sparql.engine.**main.iterator.**
QueryIterService.nextStage(**QueryIterService.java:53)
at
com.hp.hpl.jena.sparql.engine.**iterator.QueryIterRepeatApply.**
makeNextStage(**QueryIterRepeatApply.java:113)
at
com.hp.hpl.jena.sparql.engine.**iterator.QueryIterRepeatApply.**
hasNextBinding(**QueryIterRepeatApply.java:65)
at
com.hp.hpl.jena.sparql.engine.**iterator.QueryIteratorBase.**
hasNext(QueryIteratorBase.**java:108)
at
com.hp.hpl.jena.sparql.engine.**iterator.QueryIterConvert.**
hasNextBinding(**QueryIterConvert.java:65)
at
com.hp.hpl.jena.sparql.engine.**iterator.QueryIteratorBase.**
hasNext(QueryIteratorBase.**java:108)
at
com.hp.hpl.jena.sparql.engine.**iterator.QueryIteratorWrapper.**
hasNextBinding(**QueryIteratorWrapper.java:40)
at
com.hp.hpl.jena.sparql.engine.**iterator.QueryIteratorBase.**
hasNext(QueryIteratorBase.**java:108)
at
com.hp.hpl.jena.sparql.engine.**iterator.QueryIteratorWrapper.**
hasNextBinding(**QueryIteratorWrapper.java:40)
at
com.hp.hpl.jena.sparql.engine.**iterator.QueryIteratorBase.**
hasNext(QueryIteratorBase.**java:108)
at
com.hp.hpl.jena.sparql.engine.**ResultSetStream.hasNext(**
ResultSetStream.java:72)
at Query.main(Query.java:42)
Caused by: HttpException: 500 Server Error
at
com.hp.hpl.jena.sparql.engine.**http.HttpQuery.execCommon(**
HttpQuery.java:320)
... 15 more
How could I get all the results?
Regards,
Regis.
On Sun, Apr 8, 2012 at 7:38 AM, Andy Seaborne<[email protected]> wrote:
On 08/04/12 01:30, Regis Pires Magalhăes wrote:
I've got a sucessful result when executing *outside* our intranet
(LAN).
All the 43016 results were returned.
It seems that there are so many requests (when executing in the Local
Network) that the server or the client can not properly
handle them. Suggestions? I could not receive more than 5000 results
when
querying inside the LAN. Results come much faster, but the exception
always
occurs.
Regis.
Is it the same exception? NoRouteToHostException isn't usually a sign
of
overload - it's a sign that the JVM inside the firewall can't contact
the
machine outside. Your proxy may be doing funny stuff though.
Do you have proxy settings given to the JVM?it should work.
-Dhttp.proxyHost=...
-Dhttp.proxyPort=...
or
-DsocksProxyHost=...
-DsocksProxyPort=...
Andy
2012/4/7 Regis Pires Magalhăes<regispiresmag@gmail.****com<
[email protected]>
I get a "java.net.****NoRouteToHostException: Cannot assign requested
address"
exception when I execute que following code:
import com.hp.hpl.jena.query.****QueryExecution;
import com.hp.hpl.jena.query.****QueryExecutionFactory;
import com.hp.hpl.jena.query.****QueryFactory;
import com.hp.hpl.jena.query.****QuerySolution;
import com.hp.hpl.jena.query.****ResultSet;
import com.hp.hpl.jena.rdf.model.****ModelFactory;
import com.hp.hpl.jena.sparql.core.****DatasetImpl;
public class Query {
public static void main(String[] args) {
String query = "PREFIX owl:<
http://www.w3.org/2002/07/owl#****<http://www.w3.org/2002/07/owl#**>>\n"
+
"PREFIX geopos:<
http://www.w3.org/2003/01/geo/****wgs84_pos#<http://www.w3.org/2003/01/geo/**wgs84_pos#>
<http://www.w3.**org/2003/01/geo/wgs84_pos#<http://www.w3.org/2003/01/geo/wgs84_pos#>
\**n"
+
"select ?s ?lat ?long where {\n" +
" SERVICE<
http://fuseki.mooo.com/lgd-****dbpedia/sparql<http://fuseki.mooo.com/lgd-**dbpedia/sparql>
<http://fuseki.**mooo.com/lgd-dbpedia/sparql<http://fuseki.mooo.com/lgd-dbpedia/sparql>
{\n" +
" select * where {\n" +
" ?s owl:sameAs ?geo .\n" +
" FILTER
regex(str(?geo),\"dbpedia\")\****n" +
" } \n" +
" }\n" +
" SERVICE<
http://fuseki.mooo.com/****dbpedia-geo/sparql<http://fuseki.mooo.com/**dbpedia-geo/sparql>
<http://**fuseki.mooo.com/dbpedia-geo/**sparql<http://fuseki.mooo.com/dbpedia-geo/sparql>
{\n" +
" select * where{\n" +
" ?geo geopos:lat ?lat ;\n" +
" geopos:long ?long .\n" +
" } \n" +
" } \n" +
"} \n";
long start = System.currentTimeMillis();
QueryExecution exec =
QueryExecutionFactory.create(****QueryFactory.create(query),
new
DatasetImpl(ModelFactory.****createDefaultModel()));
ResultSet rs = exec.execSelect();
int n = 0;
while (rs.hasNext()) {
rs.next();
System.out.print(++n + " ");
}
long time = System.currentTimeMillis() - start;
System.out.println(n + " results found in " + time +
"ms.");
}
}
The same query runs perfectly when a do limit in the first SERVICE
select:
String query = "PREFIX owl:<
http://www.w3.org/2002/07/owl#****<http://www.w3.org/2002/07/owl#**>>\n"
+
"PREFIX geopos:<
http://www.w3.org/2003/01/geo/****wgs84_pos#<http://www.w3.org/2003/01/geo/**wgs84_pos#>
<http://www.w3.**org/2003/01/geo/wgs84_pos#<http://www.w3.org/2003/01/geo/wgs84_pos#>
\**n"
+
"select ?s ?lat ?long where {\n" +
" SERVICE<
http://fuseki.mooo.com/lgd-****dbpedia/sparql<http://fuseki.mooo.com/lgd-**dbpedia/sparql>
<http://fuseki.**mooo.com/lgd-dbpedia/sparql<http://fuseki.mooo.com/lgd-dbpedia/sparql>
{\n" +
" select * where {\n" +
" ?s owl:sameAs ?geo .\n" +
" FILTER
regex(str(?geo),\"dbpedia\")\****n" +
" } *limit 20000* \n" +
" }\n" +
" service<
http://fuseki.mooo.com/****dbpedia-geo/sparql<http://fuseki.mooo.com/**dbpedia-geo/sparql>
<http://**fuseki.mooo.com/dbpedia-geo/**sparql<http://fuseki.mooo.com/dbpedia-geo/sparql>
{\n" +
" select * where{\n" +
" ?geo geopos:lat ?lat ;\n" +
" geopos:long ?long .\n" +
" } \n" +
" } \n" +
"} \n";
What can I do to fix the problem? It always occurs when the query is
run,
even when the Fuseki Sparql Endpoint is running locally.
I did not observe any problem related to the network. The problem
persists
when the code (query) is executed in different hosts.
I am using Jena classes from "fuseki-server.jar" in the client
classpath.
I've also tested different Fuseki jar versions: 0.2.1, 0.2.2.
The full stack trace is this:
Exception in thread "main" HttpException:
java.net.****NoRouteToHostException: Cannot assign requested address:
java.net.****NoRouteToHostException: Cannot assign requested address
at
com.hp.hpl.jena.sparql.engine.****http.HttpQuery.execGet(**
HttpQuery.java:216)
at com.hp.hpl.jena.sparql.engine.****http.HttpQuery.exec(**
HttpQuery.**
java:157)
at com.hp.hpl.jena.sparql.engine.****http.Service.exec(Service.**
java:71)
at
com.hp.hpl.jena.sparql.engine.****main.iterator.**
QueryIterService.nextStage(****QueryIterService.java:53)
at
com.hp.hpl.jena.sparql.engine.****iterator.**QueryIterRepeatApply.**
makeNextStage(****QueryIterRepeatApply.java:113)
at
com.hp.hpl.jena.sparql.engine.****iterator.**QueryIterRepeatApply.**
hasNextBinding(****QueryIterRepeatApply.java:65)
at
com.hp.hpl.jena.sparql.engine.****iterator.QueryIteratorBase.****
hasNext(QueryIteratorBase.****java:108)
at
com.hp.hpl.jena.sparql.engine.****iterator.QueryIterConvert.**
hasNextBinding(****QueryIterConvert.java:65)
at
com.hp.hpl.jena.sparql.engine.****iterator.QueryIteratorBase.****
hasNext(QueryIteratorBase.****java:108)
at
com.hp.hpl.jena.sparql.engine.****iterator.**QueryIteratorWrapper.**
hasNextBinding(****QueryIteratorWrapper.java:40)
at
com.hp.hpl.jena.sparql.engine.****iterator.QueryIteratorBase.****
hasNext(QueryIteratorBase.****java:108)
at
com.hp.hpl.jena.sparql.engine.****iterator.**QueryIteratorWrapper.**
hasNextBinding(****QueryIteratorWrapper.java:40)
at
com.hp.hpl.jena.sparql.engine.****iterator.QueryIteratorBase.****
hasNext(QueryIteratorBase.****java:108)
at
com.hp.hpl.jena.sparql.engine.****ResultSetStream.hasNext(**
ResultSetStream.java:72)
at Query.main(Query.java:34)
Caused by: java.net.****NoRouteToHostException: Cannot assign
requested
address
at java.net.PlainSocketImpl.****socketConnect(Native Method)
at
java.net.****AbstractPlainSocketImpl.****doConnect(**
AbstractPlainSocketImpl.java:****327)
at
java.net.****AbstractPlainSocketImpl.****connectToAddress(**
AbstractPlainSocketImpl.java:****193)
at
java.net.****AbstractPlainSocketImpl.****connect(**
AbstractPlainSocketImpl.java:****180)
at java.net.SocksSocketImpl.****connect(SocksSocketImpl.java:****384)
at java.net.Socket.connect(****Socket.java:546)
at java.net.Socket.connect(****Socket.java:495)
at sun.net.NetworkClient.****doConnect(NetworkClient.java:****178)
at sun.net.www.http.HttpClient.****openServer(HttpClient.java:****
409)
at sun.net.www.http.HttpClient.****openServer(HttpClient.java:****
530)
at sun.net.www.http.HttpClient.<****init>(HttpClient.java:240)
at sun.net.www.http.HttpClient.****New(HttpClient.java:321)
at sun.net.www.http.HttpClient.****New(HttpClient.java:338)
at
sun.net.www.protocol.http.****HttpURLConnection.****
getNewHttpClient(**
HttpURLConnection.java:935)
at
sun.net.www.protocol.http.****HttpURLConnection.****plainConnect(**
HttpURLConnection.java:876)
at
sun.net.www.protocol.http.****HttpURLConnection.connect(**
HttpURLConnection.java:801)
at
com.hp.hpl.jena.sparql.engine.****http.HttpQuery.execGet(**
HttpQuery.java:199)
... 14 more
Best regards,
Regis.
