Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-16 Thread Jody Garnett
Back in the thick of things again, this thread was a bit too technical to keep up with while on the road. If I had to quickly do what you are describing I would set up each sql view as you describe, and then use the pre generalized datastore to swap between them based on scale. I am not *sure* the

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-13 Thread Jody Garnett
Hi Martin, I may have a related idea I am looking to implement for GeoTools 14.x - I would like to make our query system take expressions (rather than just property names). I had originally intended this for a "java views feature". The other thing I would like to do for make it easier to match up

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-13 Thread Martin Davis
Ok, I misunderstood the scope of your comment. Anything that will ultimately provide the needed flexibility sounds fine. On Fri, Mar 13, 2015 at 9:39 AM, Andrea Aime wrote: > On Fri, Mar 13, 2015 at 5:23 PM, Martin Davis wrote: > >> So does this mean that there is no way to provide the ability

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-13 Thread Andrea Aime
On Fri, Mar 13, 2015 at 5:23 PM, Martin Davis wrote: > So does this mean that there is no way to provide the ability for custom > geometry transformation on the database side? > > I think during the thread many ways to do this kind of customization were offered, that would not break general filte

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-13 Thread Martin Davis
So does this mean that there is no way to provide the ability for custom geometry transformation on the database side? On Thu, Mar 12, 2015 at 9:56 PM, Andrea Aime wrote: > Martin, > I see what you mean, and it would work in your particular case, but it > does not make > sense in general, please

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Andrea Aime
Martin, I see what you mean, and it would work in your particular case, but it does not make sense in general, please review how sql views work, under the notion of a general mechanism (as a data source that can be queried), not under the lenses of your specific wms generalization use case, and it

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Andrea Aime
On Fri, Mar 13, 2015 at 3:20 AM, Martin Davis wrote: > Hi, >>> >>> >>> About crazy talks, what our hardcore Oracle developers do first when >>> they face problems with efficiency is to start writing some PL/SQL >>> procedures which they then tune until they are satisfied (of when we say >>> that

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Martin Davis
> > > > >> >> To take another slant on this, I realized that the source of the problem >> is that for SQL views the "custom SQL" is wrapped in parens before the WMS >> bbox filter condition is appended to it. This is what's causing the DB to >> complain that the geometry column expression doesn't

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Martin Davis
On Thu, Mar 12, 2015 at 3:27 PM, Andrea Aime wrote: > On Thu, Mar 12, 2015 at 7:00 PM, Martin Davis wrote: > >> Now who's talking crazy! :) >> >> +0 on using a full templating language. Seems perhaps a bit of overkill >> for the immediate need? The "macro language" already discussed would >>

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Martin Davis
> > Hi, >> >> >> About crazy talks, what our hardcore Oracle developers do first when >> they face problems with efficiency is to start writing some PL/SQL >> procedures which they then tune until they are satisfied (of when we say >> that we have paid enough). Unfortunately Geoserver can't utiliz

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Rahkonen Jukka (MML)
Hi, Good to know, we must make some tests. All that we have in WFS is implemented before SQL Views were supported or made with the same recipes later. Are here any users who utilize stored procedures listening and willing to tell about how it goes? -Jukka- A

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Andrea Aime
On Thu, Mar 12, 2015 at 7:00 PM, Martin Davis wrote: > Now who's talking crazy! :) > > +0 on using a full templating language. Seems perhaps a bit of overkill > for the immediate need? The "macro language" already discussed would > suffice for the immediate need I think? > > The thing is, we h

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Andrea Aime
On Thu, Mar 12, 2015 at 7:40 PM, Rahkonen Jukka (MML) < jukka.rahko...@maanmittauslaitos.fi> wrote: > Hi, > > > About crazy talks, what our hardcore Oracle developers do first when > they face problems with efficiency is to start writing some PL/SQL > procedures which they then tune until they a

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Martin Davis
I think the scheme being discussed should support that nicely. It would be great to support allowing an arbitrary SQL expression to compute the geometry returned for WMS usage. That could be a custom Oracle function which performed some kind of special purpose simplification (if not already provi

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Rahkonen Jukka (MML)
Hi, About crazy talks, what our hardcore Oracle developers do first when they face problems with efficiency is to start writing some PL/SQL procedures which they then tune until they are satisfied (of when we say that we have paid enough). Unfortunately Geoserver can't utilize those fancy crea

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Martin Davis
Now who's talking crazy! :) +0 on using a full templating language. Seems perhaps a bit of overkill for the immediate need? The "macro language" already discussed would suffice for the immediate need I think? Whatever the syntax used, it seems like the solution has to: - allow altering the ge

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Martin Davis
Thanks for the idea, Ben. I did look at this briefly, but thought I'd pursue the simpler approach first. Although auto-simplification would be much nicer if possible, since it would be more general purpose and give a nicer visual appearance. I think I've seen or heard that the Oracle SDO_UTIL.SI

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Rahkonen Jukka (MML)
Hi, I am not sure but perhaps you could fool Oracle by inserting the point geometry of the polygon into the SDO_POINT array of SDO_GEOMETRY. Oracle documentation says that if SDO_ELEM_INFO and SDO_ORDINATES arrays are not both null then SDO_POINT will be skipped. However, it would be so nice t

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-12 Thread Andrea Aime
On Tue, Mar 10, 2015 at 7:00 PM, Martin Davis wrote: > We're working on tuning a GeoServer implementation over an Oracle SDO > database. We think we are seeing that the network latency of queries is the > hotspot in the performance of GeoServer render requests. > > It seems that for some situatio

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Andrea Aime
On Thu, Mar 12, 2015 at 1:15 AM, Martin Davis wrote: > Uuuh... the result might become very confusing... in the simple case you >> might want to just change a function call, >> in the complex one you might have to rewrite half of the query. >> > > Well, let's not let the best be the enemy of the

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Martin Davis
On Wed, Mar 11, 2015 at 5:00 PM, Andrea Aime wrote: > On Thu, Mar 12, 2015 at 12:18 AM, Martin Davis wrote: > >> 1. allow specifying a SQL transformation expression to be invoked only when the query is being made for WMS rendering >>> >>> What I proposed previously (the bbox thing) has

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Andrea Aime
On Thu, Mar 12, 2015 at 12:18 AM, Martin Davis wrote: > 1. allow specifying a SQL transformation expression to be invoked only >>> when the query is being made for WMS rendering >>> >> >> What I proposed previously (the bbox thing) has no WMS specific >> limitations, and should still allow you to

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Martin Davis
On Wed, Mar 11, 2015 at 4:03 PM, Andrea Aime wrote: > On Wed, Mar 11, 2015 at 11:35 PM, Martin Davis wrote: >> >> Let me put it another way. I'm proposing a way of giving the user the >> ability to define how geometries are transformed on the DB side to make >> them more efficient for WMS rende

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread David Zwiers
Out of the box idea, and I realize this is a change the the oracle datastore ... but couldn't you change the Oracle datastore to check if an index exists in the data dictionary for the field you want returned before emitting the SQL from a BBOX? When the index exists for the desired column all is

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Andrea Aime
On Wed, Mar 11, 2015 at 11:35 PM, Martin Davis wrote: > > Let me put it another way. I'm proposing a way of giving the user the > ability to define how geometries are transformed on the DB side to make > them more efficient for WMS rendering (while still making the raw data > available for WFS).

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Martin Davis
> > > >> >> >> >> To think of this in a more general way, the issue under discussion >> reflects the fundamental dichotomy between WMS (rendering) queries and WFS >> (data) queries. For WMS queries it's useful to allow lots of scope to >> manipulate the data to reduce size, etc, whereas for WFS j

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Martin Davis
> >> In the extreme case I can see that the user might specify an entirely >> different query in the SQLView for WMS usage. That's a bit extreme and >> error prone, though. So dialing that back a bit we get closer to what >> we're talking about - where various parts of the query can be altered fo

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Martin Davis
The reason for the { } is to allow for multiple comma-separated parameters in the future. If commas are used as separators, then something needs to delimit any SQL code provided. But I realize this is looking pretty far ahead. Formally, the syntax is: subst := % name [ ( paramList ) ] % paramLi

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Andrea Aime
On Wed, Mar 11, 2015 at 8:12 PM, Martin Davis wrote: > > > On Tue, Mar 10, 2015 at 4:23 PM, Andrea Aime > wrote: > >> On Tue, Mar 10, 2015 at 11:07 PM, Martin Davis >> wrote: >> >>> Not sure I get all the constraints around this 100%, but generally the >>> idea sounds good. I like the idea of

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Martin Davis
On Tue, Mar 10, 2015 at 4:23 PM, Andrea Aime wrote: > On Tue, Mar 10, 2015 at 11:07 PM, Martin Davis wrote: > >> Not sure I get all the constraints around this 100%, but generally the >> idea sounds good. I like the idea of being able to take control of the >> filter condition SQL being used (w

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-11 Thread Rahkonen Jukka (MML)
Hi, If simplification on Oracle side starts to interest someday, this ticket follows what is going on in QGIS: https://hub.qgis.org/issues/12298 -Jukka Rahkonen- Andrea Aime wrote: On Tue, Mar 10, 2015 at 11:07 PM, Martin Davis mailto:mtncl...@gmail.com>> wrote: Not sure I get all the constrai

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Andrea Aime
On Tue, Mar 10, 2015 at 11:07 PM, Martin Davis wrote: > Not sure I get all the constraints around this 100%, but generally the > idea sounds good. I like the idea of being able to take control of the > filter condition SQL being used (which pretty much requires access to the > $bbox parameter).

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Martin Davis
Not sure I get all the constraints around this 100%, but generally the idea sounds good. I like the idea of being able to take control of the filter condition SQL being used (which pretty much requires access to the $bbox parameter). And it's a good idea to make it optional, so it isn't always ex

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Andrea Aime
On Tue, Mar 10, 2015 at 7:45 PM, Martin Davis wrote: > When you say "encode the search queries accordingly" do you mean leave off > the spatial filter entirely? That won't work in my case, there is too much > data to not use an index. I think what I want is a way to force the use of > another g

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Martin Davis
I'm pretty sure the SDO_POINT field is only populated if the geometry is in fact a point (this is to reduce the size of the geometry object). So that won't work in our case, since the data is polygons. We could indeed create a new column, populated via a trigger, and create an index on it (I thin

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Rahkonen Jukka (MML)
SDO_GEOMETRY is built this way: CREATE TYPE sdo_geometry AS OBJECT ( SDO_GTYPE NUMBER, SDO_SRID NUMBER, SDO_POINT SDO_POINT_TYPE, SDO_ELEM_INFO SDO_ELEM_INFO_ARRAY, SDO_ORDINATES SDO_ORDINATE_ARRAY); You can query "select sdo_point from geometry_column" if SDO_POINT has data in it. Like

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Martin Davis
Do you mean creating a new SDO_GEOMETRY on the fly in the query, but using only a single point from the original geometry? Wouldn't Oracle still detect that the "virtual geometry" doesn't have an index? Or am I missing something? Can you give an example of what you mean? On Tue, Mar 10, 2015 at

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Martin Davis
When you say "encode the search queries accordingly" do you mean leave off the spatial filter entirely? That won't work in my case, there is too much data to not use an index. I think what I want is a way to force the use of another geometry column (but this seems like it would have to be a bit o

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Andrea Aime
On Tue, Mar 10, 2015 at 7:27 PM, Martin Davis wrote: > Well, good that there's an issue for this. > > I don't understand how the SQL can be changed to not require a spatial > index? What I'm suggesting is a way to use the spatial filter on the > underlying geometry, but return a different geomet

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Martin Davis
Because I was hoping that GeoServer would make use of the GEOMETRY column in the spatial filter. I tried NOT including it first of all, and that didn't work, since the filter was applied on the PT field, causing the no-index error. Unfortunately, GeoServer is too smart - if the SLD is based on th

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Rahkonen Jukka (MML)
Hi, Why do you select geometry and centroid into SQL view if you are only interested in the centroid? "select GEOMETRY, SDO_GEOM.SDO_CENTROID(GEOMETRY, 1) PT from CUT_BLOCK_SP " -Jukka Rahkonen- Martin Davis wrote: > We're working on tuning a GeoServer impl

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Martin Davis
Well, good that there's an issue for this. I don't understand how the SQL can be changed to not require a spatial index? What I'm suggesting is a way to use the spatial filter on the underlying geometry, but return a different geometry column. Not sure if this is possible in the current GS archi

Re: [Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Benjamin Trigona-Harany
Hi Martin, PostGIS stores now (as of GeoServer 2.5) support an option to automatically use ST_Simplify on geometries, which would probably do exactly what you want if the functionality could be extended to Oracle stores. However, I do see Andrea's comment on simplification in Oracle was "Now, this

[Geoserver-users] Reducing query data size in Oracle

2015-03-10 Thread Martin Davis
We're working on tuning a GeoServer implementation over an Oracle SDO database. We think we are seeing that the network latency of queries is the hotspot in the performance of GeoServer render requests. It seems that for some situations one way to reduce the data being pushed over the wire is to r