Re: [Geotools-devel] New community module: reshape feature source/store
On Mon, Aug 6, 2012 at 10:17 AM, wrote: > Looks good, +1 for the module. > > I see another possible future use case, access control for individual > attributes. It should be possible to define such a feature hiding some > attributes and use this feature for access control. > > Another possibility could be adding converters returning null if a user has > no access privilege for an attribute. Btw, this functionality is already available in the GeoServer security subsystem, but to drive it today you have to use either GeoFence (previously known as GeoRepository) or GeoShield Cheers Andrea -- == Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via Poggio alle Viti 1187 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 962313 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] New community module: reshape feature source/store
Looks good, +1 for the module. I see another possible future use case, access control for individual attributes. It should be possible to define such a feature hiding some attributes and use this feature for access control. Another possibility could be adding converters returning null if a user has no access privilege for an attribute. Christian This message was sent using IMP, the Internet Messaging Program. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] New community module: reshape feature source/store
On Mon, Aug 6, 2012 at 5:53 AM, Gabriel Roldan wrote: > Hi Andrea, > > I think this makes for a very welcomed contribution. > > Of particular interest to me is the ability to "reshape" shapefiles. > It's actually something that's been rounding my mind for the last week > anticipating its need for another project. What do you think about the > idea of having a (yet another) shapefile sidecar file that somehow > defines the schema of the shapefile? The use case is exporting a > feature type to shapefile without loosing type accuracy and/or > attribute names. I see. Well, sure, why not, I guess we can add this ability inside the ShapefileDataStoreFactory, if the sidecar file is there it would first build the plain shapefile and then wrap it. Hm... actually there is a bit of a show stopper though. The wrapper is meant for featuresource/featurestore, it actually builds a data store too but right now it does not have the full capabilities that a store should have, in particular the readers and writers method cannot be implemented against the feature source abilities because they take a Transaction per method, while the feature source/store have a transaction as a field. Now that I think about it though, it's just a matter of grabbing the DataStore off the FeatureSource and fall back on those methods. Long story short, to get where you want I guess the following is needed: - fix the reshape store so that the missing methods are filled in - push the module into supported land, so that shapefile can depend on it - devise a file format that can hold the transforming definitions, use it as a sidecar for the shapefile Next weekend I'll merge the store into the official GeoTools repo Cheers Andrea -- == Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via Poggio alle Viti 1187 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 962313 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] New community module: reshape feature source/store
On Mon, Aug 6, 2012 at 5:15 AM, Jody Garnett wrote: > If I can recommend calling it "transform" as the name "reshape" has caused > no end of confusion. Indeed you will find the current process called > "transform". Cool, will do that > +1 on the new module however! > > I am very impressed with the read and write support, nicely done. Thanks Cheers Andrea -- == Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via Poggio alle Viti 1187 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 962313 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] New community module: reshape feature source/store
Hi Andrea,
I think this makes for a very welcomed contribution.
Of particular interest to me is the ability to "reshape" shapefiles.
It's actually something that's been rounding my mind for the last week
anticipating its need for another project. What do you think about the
idea of having a (yet another) shapefile sidecar file that somehow
defines the schema of the shapefile? The use case is exporting a
feature type to shapefile without loosing type accuracy and/or
attribute names.
In any case, +1 for the new module.
Cheers,
Gabriel
On Mon, Aug 6, 2012 at 12:15 AM, Jody Garnett wrote:
> If I can recommend calling it "transform" as the name "reshape" has caused
> no end of confusion. Indeed you will find the current process called
> "transform".
>
> +1 on the new module however!
>
> I am very impressed with the read and write support, nicely done.
> --
> Jody Garnett
>
> On Monday, 6 August 2012 at 2:37 AM, Andrea Aime wrote:
>
> Hi all,
> during the last few weekends I've been working on a reshaping
> SimpleFeatureSource/SimpleFeatureStore wrapper:
> https://github.com/aaime/geotools/tree/reshape/modules/unsupported/reshape/src/main/java/org/geotools/data/reshape
> https://github.com/aaime/geotools/tree/reshape/modules/unsupported/reshape/src/test/java/org/geotools/data/reshape
>
> The work is inspired and based on Jody's reshape process, but working
> at the FeatureSource/FeatureStore level
> is more feature rich. Short feature summary:
> * wrap any source/store and rename, type cast or hide any existing
> attribute, plus the ability to add new
> attributes computed via expressions
> * the above works also around stores, and preserves their writability
> within some limitations (more on this later)
> * full support for paging and sorting, either on native properties or
> computed ones
> * fully streaming in both directions (read and write)
> * around 70% code coverage
> * better heuristics to compute/guess the type of each attribute given
> only the start schema and the expressions
> (though it's possible to force a specific target type by hand)
>
> The set of classes is relatively large, but only one is visible, a
> factory that builds the wrapper given the
> source, the target type name and the list of Definitions (which
> contain the definition of each new attribute).
> Here is an example of how it can be used:
>
> List definitions = new ArrayList();
> definitions.add(new Definition("geom",
> ECQL.toExpression("buffer(the_geom, 1)")));
> definitions.add(new Definition("name",
> ECQL.toExpression("strToLowercase(state_name)")));
> definitions.add(new Definition("total",
> ECQL.toExpression("male + female")));
> definitions.add(new Definition("logp",
> ECQL.toExpression("log(persons)")));
>
> SimpleFeatureSource reshaped = ReshapeFactory.reshape(STATES,
> "bstates", definitions);
>
> The set of use cases I see for this tool:
> * rename ugly attribute names coming from the database (shapefile or
> Oracle are two common sources
> of butt ugly names) without losing the ability to perform
> transactions --> this would allow to run WFS CITE
> tests against Oracle for example
> * add new attributes that were not available in the source
> * cast the type of an attribute to something else (for example, a
> String with WKT could be turned into a Geometry
> via converters, which would allow to run WFS 1.1 CITE tests against
> shapefiles too)
> * replace the geometryless store, this would require a new function
> point(x, y) and a way to invert that function
> within filters so that we can turn, say, bbox into a set of
> predicates against x and y, which is something that
> could be done within the simplifying filter visitor
> * replace the retyping data store in GeoServer with something at the
> same time more general and more
> efficient (since it wraps directly a feature source, not a store)
>
> Keeping writability is an interesting subject. So far I've been a bit
> sloppy, things can definitely be improved.
> What I have so far is a notion of Definition inversion:
>
> public List inverse();
>
> The current code is simple and can only invert renaming, so only
> attributes that are not really computed
> can be inverted and thus be written on.
> This could be improved upon in some ways, for example:
> * have a general concept of invertible function
> * have a pluggable way to create "definition inverters" that can be
> added at runtime
> * for some limited cases, try to invert algebraic expressions of a
> single variable (e.g. a = b + 10)
>
> So far I've kept it simple, the above method can be overridden, so an
> implementor can decide to roll
> her own inversion logic.
>
> Writability control is also particularly loose right now, basically if
> we can invert at least one definition
> we come up with a writable wrapper, otherwise we return a read only one.
> I guess at the very least one should check if all the mandatory fields
> can be inverted... which brings
> an interesting riddle, what if one is not ava
Re: [Geotools-devel] New community module: reshape feature source/store
If I can recommend calling it "transform" as the name "reshape" has caused no
end of confusion. Indeed you will find the current process called "transform".
+1 on the new module however!
I am very impressed with the read and write support, nicely done.
--
Jody Garnett
On Monday, 6 August 2012 at 2:37 AM, Andrea Aime wrote:
> Hi all,
> during the last few weekends I've been working on a reshaping
> SimpleFeatureSource/SimpleFeatureStore wrapper:
> https://github.com/aaime/geotools/tree/reshape/modules/unsupported/reshape/src/main/java/org/geotools/data/reshape
> https://github.com/aaime/geotools/tree/reshape/modules/unsupported/reshape/src/test/java/org/geotools/data/reshape
>
> The work is inspired and based on Jody's reshape process, but working
> at the FeatureSource/FeatureStore level
> is more feature rich. Short feature summary:
> * wrap any source/store and rename, type cast or hide any existing
> attribute, plus the ability to add new
> attributes computed via expressions
> * the above works also around stores, and preserves their writability
> within some limitations (more on this later)
> * full support for paging and sorting, either on native properties or
> computed ones
> * fully streaming in both directions (read and write)
> * around 70% code coverage
> * better heuristics to compute/guess the type of each attribute given
> only the start schema and the expressions
> (though it's possible to force a specific target type by hand)
>
> The set of classes is relatively large, but only one is visible, a
> factory that builds the wrapper given the
> source, the target type name and the list of Definitions (which
> contain the definition of each new attribute).
> Here is an example of how it can be used:
>
> List definitions = new ArrayList();
> definitions.add(new Definition("geom",
> ECQL.toExpression("buffer(the_geom, 1)")));
> definitions.add(new Definition("name",
> ECQL.toExpression("strToLowercase(state_name)")));
> definitions.add(new Definition("total",
> ECQL.toExpression("male + female")));
> definitions.add(new Definition("logp",
> ECQL.toExpression("log(persons)")));
>
> SimpleFeatureSource reshaped = ReshapeFactory.reshape(STATES,
> "bstates", definitions);
>
> The set of use cases I see for this tool:
> * rename ugly attribute names coming from the database (shapefile or
> Oracle are two common sources
> of butt ugly names) without losing the ability to perform
> transactions --> this would allow to run WFS CITE
> tests against Oracle for example
> * add new attributes that were not available in the source
> * cast the type of an attribute to something else (for example, a
> String with WKT could be turned into a Geometry
> via converters, which would allow to run WFS 1.1 CITE tests against
> shapefiles too)
> * replace the geometryless store, this would require a new function
> point(x, y) and a way to invert that function
> within filters so that we can turn, say, bbox into a set of
> predicates against x and y, which is something that
> could be done within the simplifying filter visitor
> * replace the retyping data store in GeoServer with something at the
> same time more general and more
> efficient (since it wraps directly a feature source, not a store)
>
> Keeping writability is an interesting subject. So far I've been a bit
> sloppy, things can definitely be improved.
> What I have so far is a notion of Definition inversion:
>
> public List inverse();
>
> The current code is simple and can only invert renaming, so only
> attributes that are not really computed
> can be inverted and thus be written on.
> This could be improved upon in some ways, for example:
> * have a general concept of invertible function
> * have a pluggable way to create "definition inverters" that can be
> added at runtime
> * for some limited cases, try to invert algebraic expressions of a
> single variable (e.g. a = b + 10)
>
> So far I've kept it simple, the above method can be overridden, so an
> implementor can decide to roll
> her own inversion logic.
>
> Writability control is also particularly loose right now, basically if
> we can invert at least one definition
> we come up with a writable wrapper, otherwise we return a read only one.
> I guess at the very least one should check if all the mandatory fields
> can be inverted... which brings
> an interesting riddle, what if one is not available, but I still want
> the wrapper to be writable?
> I guess the API could provide default values for the fields that get
> hidden and/or cannot be inverted.
>
> Feedback/opinions welcomed
>
> Ah, one more thing: on the branch I've linked above I've been pretty
> loose about commits, before
> merging into trunk I intend to squash them, so please don't fork it off
>
> Cheers
> Andrea
>
> --
> ==
> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
> more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via Poggio alle Viti 1187
[Geotools-devel] New community module: reshape feature source/store
Hi all,
during the last few weekends I've been working on a reshaping
SimpleFeatureSource/SimpleFeatureStore wrapper:
https://github.com/aaime/geotools/tree/reshape/modules/unsupported/reshape/src/main/java/org/geotools/data/reshape
https://github.com/aaime/geotools/tree/reshape/modules/unsupported/reshape/src/test/java/org/geotools/data/reshape
The work is inspired and based on Jody's reshape process, but working
at the FeatureSource/FeatureStore level
is more feature rich. Short feature summary:
* wrap any source/store and rename, type cast or hide any existing
attribute, plus the ability to add new
attributes computed via expressions
* the above works also around stores, and preserves their writability
within some limitations (more on this later)
* full support for paging and sorting, either on native properties or
computed ones
* fully streaming in both directions (read and write)
* around 70% code coverage
* better heuristics to compute/guess the type of each attribute given
only the start schema and the expressions
(though it's possible to force a specific target type by hand)
The set of classes is relatively large, but only one is visible, a
factory that builds the wrapper given the
source, the target type name and the list of Definitions (which
contain the definition of each new attribute).
Here is an example of how it can be used:
List definitions = new ArrayList();
definitions.add(new Definition("geom",
ECQL.toExpression("buffer(the_geom, 1)")));
definitions.add(new Definition("name",
ECQL.toExpression("strToLowercase(state_name)")));
definitions.add(new Definition("total",
ECQL.toExpression("male + female")));
definitions.add(new Definition("logp",
ECQL.toExpression("log(persons)")));
SimpleFeatureSource reshaped = ReshapeFactory.reshape(STATES,
"bstates", definitions);
The set of use cases I see for this tool:
* rename ugly attribute names coming from the database (shapefile or
Oracle are two common sources
of butt ugly names) without losing the ability to perform
transactions --> this would allow to run WFS CITE
tests against Oracle for example
* add new attributes that were not available in the source
* cast the type of an attribute to something else (for example, a
String with WKT could be turned into a Geometry
via converters, which would allow to run WFS 1.1 CITE tests against
shapefiles too)
* replace the geometryless store, this would require a new function
point(x, y) and a way to invert that function
within filters so that we can turn, say, bbox into a set of
predicates against x and y, which is something that
could be done within the simplifying filter visitor
* replace the retyping data store in GeoServer with something at the
same time more general and more
efficient (since it wraps directly a feature source, not a store)
Keeping writability is an interesting subject. So far I've been a bit
sloppy, things can definitely be improved.
What I have so far is a notion of Definition inversion:
public List inverse();
The current code is simple and can only invert renaming, so only
attributes that are not really computed
can be inverted and thus be written on.
This could be improved upon in some ways, for example:
* have a general concept of invertible function
* have a pluggable way to create "definition inverters" that can be
added at runtime
* for some limited cases, try to invert algebraic expressions of a
single variable (e.g. a = b + 10)
So far I've kept it simple, the above method can be overridden, so an
implementor can decide to roll
her own inversion logic.
Writability control is also particularly loose right now, basically if
we can invert at least one definition
we come up with a writable wrapper, otherwise we return a read only one.
I guess at the very least one should check if all the mandatory fields
can be inverted... which brings
an interesting riddle, what if one is not available, but I still want
the wrapper to be writable?
I guess the API could provide default values for the fields that get
hidden and/or cannot be inverted.
Feedback/opinions welcomed
Ah, one more thing: on the branch I've linked above I've been pretty
loose about commits, before
merging into trunk I intend to squash them, so please don't fork it off
Cheers
Andrea
--
==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.
==
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
---
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include
