Thanks,
Greg
On 04/12/2018 12:01, Marco Neumann wrote:
comments inline
On Mon, Dec 3, 2018 at 5:14 PM Greg Albiston <[email protected]>
wrote:
Hi Marco,
1. As mentioned this shouldn't be too difficult to support.
indeed not difficult but needs a decision
you could try with the following geonames dataset
all-geonames_lotico.ttl.gz
2. Yes, the indexing, or rather caching, is in-memory, but it is
on-demand. There shouldn't be any delay at start-up beyond what Jena
needs to do. The cost comes during query execution. The key invariant
data produced for solutions is retained for a short period of time (but
can be configured to be retained until termination). Some regularly
re-used info is always kept until termination (e.g. any spatial
reference system transformation that has been requested).
the following will create and populate the TDB dataset
./geosparql-fuseki --loopback false --rdf_file ./lm.ttl --tdb TDB1
I presume this message refers to the creation of the spatial cache /
index
6:05:46.685 INFO Applying GeoSPARQL Schema - Started
6:07:44.826 INFO Applying GeoSPARQL Schema - Completed
next time I can call TDB directly
./geosparql-fuseki --loopback false --tdb TDB1
6:08:38.665 INFO Applying GeoSPARQL Schema - Started
6:10:18.661 INFO Applying GeoSPARQL Schema - Completed
takes approximately 2m for a very small data set. the same fuseki with
tdb+jena-spatial restarts almost instantaneously even with reasonably
large
data sets (see geonames).
The main benefit of this is de-serialising geometry literals. The
spatial relations arguments are between a pair of geometry literals,
one
of which is likely to be the same in the next solution, so keeping hold
of both means in alot of cases the de-serialisation can be avoided for
one (and possibly both if still retained from a previous set of
solutions).
might be a good idea to serialize the cache object of de-serialisized
geometries to disk to speed up the boot process. maybe Andy could
assist or
even align this with tdb
The aim was to only do work that's needed, not do repeat work and to be
generally quick (i.e. rely on JTS to be optimised for quick solutions
between the geometry pairs and Jena to optimise queries). There are 24
spatial relations and about half a dozen other functions so
pre-computing every combination gets big quickly and produces data that
users might not want/use.
A rough check of most the spatial relations only requires a bounding
box
intersection or type check, so negative results can be quickly
discarded. I looked into caching and storing to file, but there just
wasn't the benefit in my use case. It took longer to load up then
execute than just execute from fresh and cache. Also, the spatial
indexes implemented by JTS aren't designed/suited for the spatial
relations. If there is a use-case that gets more benefit from
pre-computing or storing between programme execution then I'm sure it
can be adapted for, but in the context of GeoSPARQL this approach was
effective.
3. If you could send me the dataset that causes these errors then I'll
happily have a look into it.
you can use this simple list of point geometries here
http://www.lotico.com/lm.ttl.gz
this query will parse and execute
PREFIX geo: <http://www.opengis.net/ont/geosparql#>
PREFIX geof: <http://www.opengis.net/def/function/geosparql/>
SELECT ?well
WHERE {
?well <http://www.wikidata.org/entity/P625> ?geometry .
FILTER(geof:sfWithin(?geometry,"POLYGON((-10 50,2 50,2 55,-10 55,-10
50))"^^geo:wktLiteral))
} LIMIT 10
this one will parse and fail
PREFIX geo: <http://www.opengis.net/ont/geosparql#>
PREFIX geof: <http://www.opengis.net/def/function/geosparql/>
SELECT ?well
WHERE {
?well <http://www.wikidata.org/entity/P625> ?geometry .
FILTER(geof:sfWithin(?geometry,"POLYGON((-10 50,2 50,2 55,-10 55,-10
51))"^^geo:wktLiteral))
} LIMIT 10
warn/error messages
6:17:45.887 ERROR Points of LinearRing do not form a closed linestring -
Illegal WKT literal: POLYGON((-10 50,2 50,2 55,-10 55,-10 51))
6:17:45.887 WARN General exception in (<
http://www.opengis.net/def/function/geosparql/sfWithin> ?geometry
"POLYGON((-10 50,2 50,2 55,-10 55,-10 51))"^^<
http://www.opengis.net/ont/geosparql#wktLiteral>)
org.apache.jena.datatypes.DatatypeFormatException: Points of LinearRing
do
not form a closed linestring - Illegal WKT literal: POLYGON((-10 50,2
50,2
55,-10 55,-10 51))
at
io.github.galbiston.geosparql_jena.implementation.datatype.WKTDatatype.parse(WKTDatatype.java:109)
at
io.github.galbiston.geosparql_jena.implementation.GeometryWrapper.extract(GeometryWrapper.java:905)
at
io.github.galbiston.geosparql_jena.implementation.GeometryWrapper.extract(GeometryWrapper.java:834)
at
io.github.galbiston.geosparql_jena.geof.topological.GenericFilterFunction.exec(GenericFilterFunction.java:57)
at
io.github.galbiston.geosparql_jena.geof.topological.GenericFilterFunction.exec(GenericFilterFunction.java:42)
at
org.apache.jena.sparql.function.FunctionBase2.exec(FunctionBase2.java:55)
at
org.apache.jena.sparql.function.FunctionBase.exec(FunctionBase.java:63)
at
org.apache.jena.sparql.expr.E_Function.evalSpecial(E_Function.java:89)
at
org.apache.jena.sparql.expr.ExprFunctionN.eval(ExprFunctionN.java:100)
at
org.apache.jena.sparql.expr.ExprNode.isSatisfied(ExprNode.java:41)
at
org.apache.jena.sparql.engine.iterator.QueryIterFilterExpr.accept(QueryIterFilterExpr.java:49)
at
org.apache.jena.sparql.engine.iterator.QueryIterProcessBinding.hasNextBinding(QueryIterProcessBinding.java:69)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIterConvert.hasNextBinding(QueryIterConvert.java:58)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:39)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:39)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:39)
at
org.apache.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:114)
at
org.apache.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:74)
at
org.apache.jena.sparql.engine.ResultSetCheckCondition.hasNext(ResultSetCheckCondition.java:55)
at
org.apache.jena.fuseki.servlets.SPARQL_Query.executeQuery(SPARQL_Query.java:350)
at
org.apache.jena.fuseki.servlets.SPARQL_Query.execute(SPARQL_Query.java:288)
at
org.apache.jena.fuseki.servlets.SPARQL_Query.executeWithParameter(SPARQL_Query.java:242)
at
org.apache.jena.fuseki.servlets.SPARQL_Query.perform(SPARQL_Query.java:217)
at
org.apache.jena.fuseki.servlets.ActionService.executeLifecycle(ActionService.java:183)
at
org.apache.jena.fuseki.servlets.ActionService.execCommonWorker(ActionService.java:98)
at
org.apache.jena.fuseki.servlets.ActionBase.doCommon(ActionBase.java:74)
at
org.apache.jena.fuseki.servlets.FusekiFilter.doFilter(FusekiFilter.java:73)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:503)
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at
org.eclipse.jetty.io
.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io
.FillInterest.fillable(FillInterest.java:103)
at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base/java.lang.Thread.run(Thread.java:834)
4. The "geo:" prefix is the one used throughout the GeoSPARQL
documentation, so has been used for consistency when needed. The code
doesn't have a dependency on the "geo:" prefix, so there is no
requirement on the user. It would probably cause more confusion to
those
following GeoSPARQL examples to not use the "geo:" prefix when
necessary.
I know but it needs some discussion about re-purposing of prefixes here
Thanks,
Greg
On 03/12/2018 15:46, Marco Neumann wrote:
Hi Greg, ok let's do it in the dev list first.
1. indeed the picking up of lat/long is a common if not the most
common
use
case for building a spatial index. last but not least to perform a
proximity search on 2D point geometries. (I know that the ogc
recommends
a
transformation path with a sparql query to turn lat / long into a WKT
geometry datatypes maybe we could provide this as a convenient option
with
the release)
2. as far as I can see the spatial index in geosparql-jena is memory
based.
it creates additional load time during server startup. Am I missing
something here, is there a file base spatial index as well?
3. error handling is disruptive. since we are hitting the spatial
index
first during query execution I am seeing a number of unpleasant side
effects with syntactically correct sparql but semantically incorrect
spatial queries. e.g.
PREFIX geo: <http://www.opengis.net/ont/geosparql#>
PREFIX geof: <http://www.opengis.net/def/function/geosparql/>
SELECT ?well
WHERE {
?well <http://www.wikidata.org/entity/P625> ?geometry .
FILTER(geof:sfWithin(?geometry,"POLYGON((-77 38,-77 0,0 38,0 0,0
0))"^^geo:wktLiteral))
} LIMIT 10
4. The re-use of the geo: prefix really isn't your problem I know but
it
will create confusion. Wouldn't geosparql: be a better prefix for
this?
Is
the OGC now married to this prefix? It used to be
http://www.w3.org/2003/01/geo/wgs84_pos#
and there is more to come...
again thank you for working on this with your team Greg, much
appreciated.
On Mon, Dec 3, 2018 at 2:15 PM Greg Albiston <[email protected]>
wrote:
Hi Marco,
I've had a look at the doucmentation for Jena Spatial and it would
seem
the main data change is the use of the Lat/Lon pairs.
This doesn't comply with the GeoSPARQL standard so support for this
would be a Jena extension.
This could be accomodated by a property function to convert to a WKT
Point literal with WGS84/CRS84 spatial reference.
Users would then be able to use the result in query for any of the
GeoSPARQL functions.
Alternatively, the spatial relations could all have an extra property
function defined, provide the conversion and hand over to the
GeoSPARQL
equivalent property function. This wouldn't take long to integrate as
individual spatial relation property functions are very minimal.
The other item that jumps out is the Jena spatial property functions.
spatial:nearby, spatial:withinCircle, spatial:withinBox and
spatial:interesectBox all seem to be variations of Simple Features
spatial relations that are covered by GeoSPARQL. These property
functions can be incorpated for backward compatability but it's
whether
these should just be offered as the current Lat/Lon pairs or
expanded to
accept geometry literals (i.e. WKT, GML etc.)? The latter option
shouldn't be hard to provide for the same reason as above.
spatial:north, spatial:south, spatial:west and spatial:east are not
in
GeoSPARQL. Again its a question of whether these should be provided
more
generally for WKT, GML geometry literals? There might need to be a
bit
of extra work handling both geographic and planar spatial reference
systems, as Jean Spatial is only doing a spatial reference system.
I don't think it would be too difficult to support the existing Jena
Spatial functionality, at least based on the webpage
(https://jena.apache.org/documentation/query/spatial-query.html),
as an
extension to what is provided by GeoSPARQL.
Is there anything else that you were concerned about?
Thanks,
Greg
On 03/12/2018 10:53, Marco Neumann wrote:
so I've had a look at this and while I think geosparql-jena is a
very
welcomed contribution to the jena project I don't think we should
rush
with
the retirement of jena-spatial at this point as Greg's approach
will
require users to make changes to their existing data.
I will engage Greg on [email protected] again to clarify a few
things
and hopefully get more people involved in this conversation around
spatial,
geosparql and jena.
On Fri, Nov 30, 2018 at 1:23 PM Marco Neumann <
[email protected]
wrote:
how quickly can you hook geosparql into the release?
this would make lucene spatial obsolete in the next release. has
Greg
released performance benchmarks for his implementation? as I said I
will
take a look at it over the weekend when time permits.
On Fri, Nov 30, 2018 at 11:02 AM Andy Seaborne <[email protected]>
wrote:
We could retire jena-spatial immediately after 3.10.0 - given the
Lucene
change that might be smoother, one release with updated
dependencies.
If that is the way forward, I think it is (mildly) better to take
it
out
of the Fuseki/Full build in 3.10.0.
Andy
On 29/11/2018 17:00, Marco Neumann wrote:
I will have to look into that I guess since I am frequent user of
spatial
data.
why not go to 7.5? was there an incompatibility?
On Thu 29. Nov 2018 at 16:53, Andy Seaborne <[email protected]>
wrote:
Jena 3.1.0 would be around the end of the year. I'd like to make
use
of
Greg's GeoSPARQL project the "headline" item for the release
and to
retire jena-spatial in 3.10.0 as an indication of this.
Because retirement is a new process for the project, I'm sending
this
first 3.10.0 message quite early to give us discussion time.
== Retirements
We have talked about this before but not actually done anything.
See
separate thread for discussion on retirement process and for the
first
modules:
jena-spatial
jena-fuseki1
jena-csv
== Headlines
JENA-664 : GeoSPARQL support
I'd like to make use of Greg's GeoSPARQL project the "headline"
item
for
the release and to retire jena-spatial in 3.10.0 as an
indication
of
this.
JENA-1621 : Lucene upgrade to 7.4
May need to reload lucene indexes.
(e.g. the lucene index was create originally with Lucene v5.x
(prior
Jena 3.3.0). See Lucene upgrade tool.
https://lucene.apache.org/solr/guide/7_4/indexupgrader-tool.html
JENA-1623 : Fuseki security
JENA-1627 : HTTP support
https://issues.apache.org/jira/browse/JENA-1623
http://jena.staging.apache.org/documentation/fuseki2/data-access-control
== JIRA:
31 currently.
https://s.apache.org/jena-3.10.0-jira
== Updates
Only plugins. JENA-1624
surefire : 2.21.0 -> 2.22.1 (+ SUREFIRE-1588)
compiler : 3.7.0 -> 3.8.0
shade : 3.1.0 -> 3.2.0
Andy
--
---
Marco Neumann
KONA
