Re: [Geotools-devel] Idea: FeatureStream

2016-08-08 Thread Jody Garnett 
the stream() method does not necessarily pull down all the data at runtime,
the individual methods use predicates etc to cut down on what is obtained.

For me the riskiest part of this idea is making Filter implement Predicate;
because Predicates are usually typed, and our Filters are not typed.

--
Jody Garnett

On 6 August 2016 at 06:14, Jim Hughes  wrote:

> On 8/6/2016 4:41 AM, Andrea Aime wrote:
>
> On Sat, Aug 6, 2016 at 9:01 AM, Jody Garnett 
> wrote:
>
>> The sad part / danger / risk is that that our Filter interface would only
>> be one kind of predicate, and I would expect normal java developers to make
>> a query quickly in java code - which is something we cannot optimize.
>>
>
> No objections to having streams around, just make sure to properly
> document them and give warning to people
> about performance issues.
>
>
> As a suggestion, maybe it make sense to have FeatureSource.getFeatureStream()
> and FeatureSource.getFeatureStream(Filter) right next to each other.
>
> Having a method which can either push-down a query or pull back all your
> data at runtime has me worried for large datasets.
>
> Cheers,
>
> Jim
>
>
> From what I gathered in the discussion so far, it's a bit like hibernate,
> it can simplify things, but in order to use it
> efficiently one needs to understand how it works internally. Maybe
> indicate the docs examples of what will
> be efficiently sent down to the store for native translation, and what
> cannot be.
>
> Wondering, are there risks about properly closing feature iterators here?
> If you are using visitor I guess not,
> but in that case there are other potential issues with our current
> interface, like inability to bail out of the iteration early.
>
> Cheers
> Andrea
>
> --
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob: +39  339 8844549
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility  for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
> ---
>
>
>
--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Idea: FeatureStream

2016-08-06 Thread Jim Hughes 

On 8/6/2016 4:41 AM, Andrea Aime wrote:
On Sat, Aug 6, 2016 at 9:01 AM, Jody Garnett > wrote:


The sad part / danger / risk is that that our Filter interface
would only be one kind of predicate, and I would expect normal
java developers to make a query quickly in java code - which is
something we cannot optimize.


No objections to having streams around, just make sure to properly 
document them and give warning to people

about performance issues.


As a suggestion, maybe it make sense to have 
FeatureSource.getFeatureStream() and 
FeatureSource.getFeatureStream(Filter) right next to each other.


Having a method which can either push-down a query or pull back all your 
data at runtime has me worried for large datasets.


Cheers,

Jim

From what I gathered in the discussion so far, it's a bit like 
hibernate, it can simplify things, but in order to use it
efficiently one needs to understand how it works internally. Maybe 
indicate the docs examples of what will
be efficiently sent down to the store for native translation, and what 
cannot be.


Wondering, are there risks about properly closing feature iterators 
here? If you are using visitor I guess not,
but in that case there are other potential issues with our current 
interface, like inability to bail out of the iteration early.


Cheers
Andrea

--
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o 
nel/i file/s allegato/i sono da considerarsi strettamente riservate. 
Il loro utilizzo è consentito esclusivamente al destinatario del 
messaggio, per le finalità indicate nel messaggio stesso. Qualora 
riceviate questo messaggio senza esserne il destinatario, Vi preghiamo 
cortesemente di darcene notizia via e-mail e di procedere alla 
distruzione del messaggio stesso, cancellandolo dal Vostro sistema. 
Conservare il messaggio stesso, divulgarlo anche in parte, 
distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità 
diverse, costituisce comportamento contrario ai principi dettati dal 
D.Lgs. 196/2003.


The information in this message and/or attachments, is intended solely 
for the attention and use of the named addressee(s) and may be 
confidential or proprietary in nature or covered by the provisions of 
privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New 
Data Protection Code).Any use not in accord with its purpose, any 
disclosure, reproduction, copying, distribution, or either 
dissemination, either whole or partial, is strictly forbidden except 
previous formal approval of the named addressee(s). If you are not the 
intended recipient, please contact immediately the sender by 
telephone, fax or e-mail and delete the information in this message 
that has been received in error. The sender does not give any warranty 
or accept liability as the content, accuracy or completeness of sent 
messages and accepts no responsibility  for changes made after they 
were sent or for other risks which arise as a result of e-mail 
transmission, viruses, etc.



---



--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Idea: FeatureStream

2016-08-06 Thread Andrea Aime 
On Sat, Aug 6, 2016 at 9:01 AM, Jody Garnett  wrote:

> The sad part / danger / risk is that that our Filter interface would only
> be one kind of predicate, and I would expect normal java developers to make
> a query quickly in java code - which is something we cannot optimize.
>

No objections to having streams around, just make sure to properly document
them and give warning to people
about performance issues.

>From what I gathered in the discussion so far, it's a bit like hibernate,
it can simplify things, but in order to use it
efficiently one needs to understand how it works internally. Maybe indicate
the docs examples of what will
be efficiently sent down to the store for native translation, and what
cannot be.

Wondering, are there risks about properly closing feature iterators here?
If you are using visitor I guess not,
but in that case there are other potential issues with our current
interface, like inability to bail out of the iteration early.

Cheers
Andrea

-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---
--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Idea: FeatureStream

2016-08-06 Thread Jody Garnett 
Yes, the point is to capture the provided predicate, and if it is one of
our Filter objects we can pass it to the featureSource. Note this is a
Filter object, since it is an object oriented language. Any given datastore
is welcome to transform that into its own internal format for fun and
profit, SQL, CQL, ECQL, etc...

Internally I expect the stream implementation to stage a normal geotools
query and:
- use the filter as normal to request the contents, followed by the
existing visitor method to handle traversal when implementing the forEach
method.
- method such anyMatch( predicate ) can be handled quite well with a Query
with max features set to 1
- etc...

The sad part / danger / risk is that that our Filter interface would only
be one kind of predicate, and I would expect normal java developers to make
a query quickly in java code - which is something we cannot optimize.

--
Jody Garnett

On 5 August 2016 at 20:20, Jim Hughes  wrote:

> Hi Devon, Jody,
>
> This looks exciting!  For the (E)CQL filters as predicates, is there any
> chance those predicates could be passed down to the FeatureSource?  Or
> would there be different execution from...
>
> featureSource.stream.filter(predicate)
> and
> featureSource.getFeatures(predicate).stream
>
> Naively, it seems what you suggested (and my first example) would stream
> back all the features and then apply all filtering 'client' side.  If there
> are tricks to write the first and have it perform database queries
> sensibly, that might be good.
>
> If that support is only partial or is intuitive, I'd worry that we are
> giving folks a way to cause too much trouble.  On the other hand, if there
> are really clever options for pushing around computation, weaving that
> through the FeatureSource/Collection API could be really powerful.
>
> Cheers,
>
> Jim
>
>
> On 8/5/2016 7:07 PM, Jody Garnett wrote:
>
> Reviewing the methods with Devon a lot of it looked pretty straight
> forward using FeatureSource and visitor.
>
> featureSource.stream().forEach(
>f -> System.out.println( f.getId() ) );
> );
>
> Once concern I had, well Kevin had, was how to handle predicates cleanly.
> There is an overlap of functionality between Predicates and our Filter
> interface - it would be great Filter could implement Predicate (allowing
> them to be used for java streams) and allowing us to detect the use of:
>
> Filter predicate = CQL.toFilter("attName >= 5");
>
> boolean found = featureSource.stream().anyMatch( predicate );
>
> featureSource.stream().filter( predicate ).forEach(
>f -> System.out.println( f.getId() ) );
> );
>
> Collection geoms = featureSource.stream().map(
> f -> (Geometry) f.getDefaultGeometry();
> ).collect( Collectors.toList() );
>
>
>
> --
> Jody Garnett
>
> On 5 August 2016 at 14:45, Devon Tucker  wrote:
>
>> Hi all,
>>
>> Jody and I were talking today about and idea he has been agitating for a
>> while. We'd like to basically implement the Java 8 streaming interface for
>> features on top of FeatureSource or FeatureCollection. Just throwing this
>> out there for early feedback to see what people think.
>>
>> Jody thinks that much of that API can be implemented in terms of the
>> existing functionality fairly simply/efficiently while others can be
>> delegated to simpler Stream objects (created from builder utilities).
>>
>> Interested in what people think. We'll probably do a quick proposal
>> shortly.
>>
>> Cheers,
>> Devon
>>
>> 
>> --
>>
>> ___
>> GeoTools-Devel mailing list
>> GeoTools-Devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>>
>>
>
>
> --
>
>
>
> ___
> GeoTools-Devel mailing 
> listGeoTools-Devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
>
> 
> --
>
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Idea: FeatureStream

2016-08-05 Thread Jim Hughes 

Hi Devon, Jody,

This looks exciting!  For the (E)CQL filters as predicates, is there any 
chance those predicates could be passed down to the FeatureSource?  Or 
would there be different execution from...


featureSource.stream.filter(predicate)
and
featureSource.getFeatures(predicate).stream

Naively, it seems what you suggested (and my first example) would stream 
back all the features and then apply all filtering 'client' side.  If 
there are tricks to write the first and have it perform database queries 
sensibly, that might be good.


If that support is only partial or is intuitive, I'd worry that we are 
giving folks a way to cause too much trouble.  On the other hand, if 
there are really clever options for pushing around computation, weaving 
that through the FeatureSource/Collection API could be really powerful.


Cheers,

Jim

On 8/5/2016 7:07 PM, Jody Garnett wrote:
Reviewing the methods with Devon a lot of it looked pretty straight 
forward using FeatureSource and visitor.


featureSource.stream().forEach(
   f -> System.out.println( f.getId() ) );
);

Once concern I had, well Kevin had, was how to handle predicates 
cleanly. There is an overlap of functionality between Predicates and 
our Filter interface - it would be great Filter could implement 
Predicate (allowing them to be used for java streams) and allowing us 
to detect the use of:


Filter predicate = CQL.toFilter("attName >= 5");

boolean found = featureSource.stream().anyMatch( predicate );

featureSource.stream().filter( predicate ).forEach(
   f -> System.out.println( f.getId() ) );
);

Collection geoms = featureSource.stream().map(
f -> (Geometry) f.getDefaultGeometry();
).collect( Collectors.toList() );



--
Jody Garnett

On 5 August 2016 at 14:45, Devon Tucker > wrote:


Hi all,

Jody and I were talking today about and idea he has been agitating
for a while. We'd like to basically implement the Java 8 streaming
interface for features on top of FeatureSource or
FeatureCollection. Just throwing this out there for early feedback
to see what people think.

Jody thinks that much of that API can be implemented in terms of
the existing functionality fairly simply/efficiently while others
can be delegated to simpler Stream objects (created from builder
utilities).

Interested in what people think. We'll probably do a quick
proposal shortly.

Cheers,
Devon


--

___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/geotools-devel




--


___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel



--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] Idea: FeatureStream

2016-08-05 Thread Jody Garnett 
Reviewing the methods with Devon a lot of it looked pretty straight forward
using FeatureSource and visitor.

featureSource.stream().forEach(
   f -> System.out.println( f.getId() ) );
);

Once concern I had, well Kevin had, was how to handle predicates cleanly.
There is an overlap of functionality between Predicates and our Filter
interface - it would be great Filter could implement Predicate (allowing
them to be used for java streams) and allowing us to detect the use of:

Filter predicate = CQL.toFilter("attName >= 5");

boolean found = featureSource.stream().anyMatch( predicate );

featureSource.stream().filter( predicate ).forEach(
   f -> System.out.println( f.getId() ) );
);

Collection geoms = featureSource.stream().map(
f -> (Geometry) f.getDefaultGeometry();
).collect( Collectors.toList() );



--
Jody Garnett

On 5 August 2016 at 14:45, Devon Tucker  wrote:

> Hi all,
>
> Jody and I were talking today about and idea he has been agitating for a
> while. We'd like to basically implement the Java 8 streaming interface for
> features on top of FeatureSource or FeatureCollection. Just throwing this
> out there for early feedback to see what people think.
>
> Jody thinks that much of that API can be implemented in terms of the
> existing functionality fairly simply/efficiently while others can be
> delegated to simpler Stream objects (created from builder utilities).
>
> Interested in what people think. We'll probably do a quick proposal
> shortly.
>
> Cheers,
> Devon
>
>
> --
>
> ___
> GeoTools-Devel mailing list
> GeoTools-Devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>
>
--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

[Geotools-devel] Idea: FeatureStream

2016-08-05 Thread Devon Tucker 
Hi all,

Jody and I were talking today about and idea he has been agitating for a
while. We'd like to basically implement the Java 8 streaming interface for
features on top of FeatureSource or FeatureCollection. Just throwing this
out there for early feedback to see what people think.

Jody thinks that much of that API can be implemented in terms of the
existing functionality fairly simply/efficiently while others can be
delegated to simpler Stream objects (created from builder utilities).

Interested in what people think. We'll probably do a quick proposal shortly.

Cheers,
Devon
--
___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel