Re: [Geotools-devel] ImageMosaic API refactor proposal

[Jody Garnett]

>
> -1- Can you clarify this sentence?
>
> "to replace the default direct granule access with a renderable graph
> (for reprojection)"
>

Replaced with "support resampling of rasters into common CRS for mosaic
operation".


> -2- Stupid thing but in ImageMosaic I would go for add/removeRaster
> rather than *Coverage
>

Updated.

This proposal (and email thread) should be orthogonal to the RnD discussion
on setting up a mosaic with multiple projections.

This mosaic is focused on creating/managing the index used. One part I
cannot wrap my head around is the workflow behind these two methods
- createSchema(String name, CoordinateReferenceSystem crs) which creates
the FeatureType used for the index, and Collection
getPropertiesCollectors().

It seems these two methods need to be implemented together, a single
PropertyCollector may collect more than once value on examining a raster. I
am not sure how to line up these collected values with the created schema
without more information. We also have String getParameter(Prop prop)
defined, but the class Prop is not included in the proposal.

Suggestions:
- Use a list of PropertiesCollectors to provide order, which can then be
used to determine the Schema?
- Language consistency - raster / granule
- Language consistency - could we use AttributeDescriptor rather than
introduce a new thing called "Prop" ?
- Are the class responsibilities clear:
GranuleCatalogManager, GranuleCatalog, GranuleStore, MosaicHarvester, etc...

(It could also be that since I am unfamiliar with the internals being
refactored that I am just struggling with the language - so if everyone
else is okay ...)
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] On-the-fly reprojection in ImageMosaic

[Jody Garnett]

Thanks for the brain dump Andrea (and to Devon for starting this
conversation). The key reason we are reaching out to geotools-devel is the
sheer wasted disk space represented by a static "gdal command line"
solution (which no doubt would perform better).

Devon has been exploring the dynamic approach (using a resample prior to
mosaic) in a branch. We did not consider grouping granules with similar CRS
- as our unsupported/dem use cases focuses on many overlapping granules
that are sorted according to need (resolution, date, etc...).

This work will be a balancing between meeting our use-case - and leaving
the design open for other stories that are perhaps more common. Thank you
for the information on "excess granule removal" - while I have identified
it as a risk I am not sure it will be addressed in this first cut.
--
Jody Garnett

On 13 April 2016 at 01:37, Andrea Aime  wrote:

> On Tue, Apr 12, 2016 at 7:17 PM, Devon Tucker 
> wrote:
>
>> Hi all,
>>
>> We've been looking into doing on-the-fly reprojection in image mosaic to
>> handle multiple projections dynamically (kinda similar to what QGIS can
>> do). Some of the issues with this we've worked out already, but one thing
>> I've been wondering is where might be an appropriate place in the code to
>> do the actual resampling.
>>
>> My first thought would be somewhere in the RasterLayerResponse either in
>> the prepareResponse or produce methods either during or after the granules
>> are collected, but I'm definitely interested in advice/feedback on what's
>> good/appropriate.
>>
>
> Hi,
> as Simone said mixing multiple projections in the same mosaic can be
> tricky.
> Mind, I'm not exactly the expert here, Daniele would know more and I'd
> invite him to chip in and correct mistakes/fill in
> voids.
> This is also going to be a bit of a brain dump, and it has a lot of
> facets... I'll just assume by now you're familiar with
> both mosaic and coverage rendering.
>
> First of all, as you have might noticed the mosaic does not use full
> coverage readers, but imageio, and uses a single grid to
> world transform for everything (assuming the geometry maps the bbox of the
> file). This is done for performance reasons,
> reading the whole file header can take time (and sometimes you have to
> read 1-2 sidecar files, depending on the format).
>
> With multiple CRSs in the mix I believe you'll have to register the CRS of
> the granule... in the lucky case that means storing
> a EPSG number, but what about CRSs that do not have a match in the EPSG
> database?
> Similar but not same, there's the issue of images that have rotated world
> to grid transforms, which we do not handle today,
> and for which I guess one would have to store the transform in the index.
>
> Either that, or start using actual coverage readers to do the reading, and
> take a performance hit for doing so.
>
> There's also the issue that the index needs to have geometries in a single
> CRS, and the mosaic itself has to advertise
> itself as a uniform CRS object, so one has to decide how to set the target
> CRS in advance I believe.
>
> Then there is the actual mosaicking. Right now we read the data, apply an
> affine to position the image in raster space
> so that it can be mosaicked, and then mosaic the images.
> With multiple CRSs it is temping to read the image, reproject it to the
> target CRS, apply affine, and then mosaic,
> but from experience doing so might result in black/white lines, or slight
> visual mis-alignemnts, due to border effects (if two tiles that are supposed
> to touch each other are reprojected separately).
>
> Normally we avoid the issue above by collecting all granules in the
> desired sequence (applying whatever sort by is required,
> e.g., freshest on top, highest res on top), then group the sequence in
> subsets of common CRS, mosaic them, reproject the
> results, make a grand mosaic of the partial results.
>
> Now, there is another catch in this game, if the mosaic reader does not
> advertise it can reproject, the rendering machinery
> will ask stuff in its advertised native CRS and then deal with
> reprojection on its own, applying a set of extra steps known
> as "advanced projection handling" to deal with common but nasty border
> cases (dateline, polar wrapping, un-reprojectable areas, and so on).
> This means two reprojections, one from native to mosaic declared CRS, and
> then another from mosaic declared to target one...
> we have a "warp to warp" simplification in ImageWorker, but it's not going
> to work always because in general you'll have mosaic operations
> in the middle (read -> mosaic same crs -> warp to mosaic crs -> mosaic all
> -> warp to target).
> Double reprojection normally has a visible effect on quality, since you're
> resampling pixels (heavily) twice, and of course
> takes also some performance toll.
> Also, in a multi-crs mosaic you're performing reprojection internally, and
> not all 

Re: [Geotools-devel] On-the-fly reprojection in ImageMosaic

[Jody Garnett]

Thanks Simone, answers inline.

> Got a question about that, when the granules overlap we make use of the
> > attribute based sort order. Does that sample all overlapping granules in
> a
> > location - or can we think of a way to limit what is sampled based on
> sort
> > order.
>
> I am not sure I understand this, can you clarify?
>

Andrea has the right words with "excess granule removal". The use-case is
having lots of different DEM data for a region, and allowing the layer to
be configured with different sort orders (to capture priority like
resolution, or how current a dataset is).

I know the danger of premature optimization, but I expect many overlapping
granules (capturing elevation data with different times, resolutions,
accuracy).

Rethinking about my initial suggestion, things might become complex if you
> want to have also transitions at the borders.
>

I assume you mean transitions at the border between granules? I guess you
could try for a feathering effect if people wanted a smooth transition -
out of scope for now :)

I would suggest you share some more information about what you are trying
> to do so that we are a better picture before suggesting more.


Understood, we should be able to stage public sample data in the
unsupported/dem module.

>


>
> > --
> > Jody
>
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] memory datastore refactor

[Jody Garnett]

I don't mind when it gets in, guess I would like in before foss4g workshop.
On Sat, Apr 16, 2016 at 3:37 AM Andrea Aime 
wrote:

> On Sat, Apr 16, 2016 at 8:03 AM, Jody Garnett 
> wrote:
>
>> A small pull request here
>> , refactoring
>> MemoryDataStore to use each entry to store features (rather than a
>> Map>
>> The goal with ContentDataStore was to use objects, rather than Maps for
>> organization. Although this refactor is functionally identical changing to
>> objects makes the logic easier to follow.
>>
>> To enable this refactor ContentEntry is no longer a final class.
>>
>
> Is this last minute change targeted to 15-beta2?
>
> Cheers
> Andrea
>
>
> --
> ==
> GeoServer Professional Services from the experts! Visit
> http://goo.gl/it488V for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob: +39  339 8844549
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility  for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
> ---
>
-- 
--
Jody Garnett
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] memory datastore refactor

[Andrea Aime]

On Sat, Apr 16, 2016 at 8:03 AM, Jody Garnett 
wrote:

> A small pull request here ,
> refactoring MemoryDataStore to use each entry to store features (rather
> than a Map
> The goal with ContentDataStore was to use objects, rather than Maps for
> organization. Although this refactor is functionally identical changing to
> objects makes the logic easier to follow.
>
> To enable this refactor ContentEntry is no longer a final class.
>

Is this last minute change targeted to 15-beta2?

Cheers
Andrea


-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39  339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Re: [Geotools-devel] ImageMosaic API refactor proposal

[Niels Charlier]

Hi Simone,

Thanks for review! In response to your questions

(1) Jody added this note to the proposal and Devon is working on this 
part, so I think either one of them are best placed to explain this part.

(2) Okay, agreed.

Regards
Niels

On 15-04-16 10:39, Simone Giannecchini wrote:
> Ciao Niels,
> we are looking into this and comparing with some work that have in the
> pipeline which might extend the work you are doing with respect to
> harvesting.
>
> I like the general idea pretty much, I have a few quick questions/obs?
>
> -1- Can you clarify this sentence?
>
> "to replace the default direct granule access with a renderable graph
> (for reprojection)"
>
> -2- Stupid thing but in ImageMosaic I would go for add/removeRaster
> rather than *Coverage
>
> I will talk to Andrea and Daniele to add more.
>
> That said, it might be worth to have a look at the code when it start to 
> settle.
>
>
>
> Regards,
> Simone Giannecchini
> ==
> GeoServer Professional Services from the experts!
> Visit http://goo.gl/it488V for more information.
> ==
> Ing. Simone Giannecchini
> @simogeo
> Founder/Director
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax: +39 0584 1660272
> mob:   +39  333 8128928
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> ---
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate.
> Il loro utilizzo è consentito esclusivamente al destinatario del
> messaggio, per le finalità indicate nel messaggio stesso. Qualora
> riceviate questo messaggio senza esserne il destinatario, Vi preghiamo
> cortesemente di darcene notizia via e-mail e di procedere alla
> distruzione del messaggio stesso, cancellandolo dal Vostro sistema.
> Conservare il messaggio stesso, divulgarlo anche in parte,
> distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità
> diverse, costituisce comportamento contrario ai principi dettati dal
> D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely
> for the attention and use of the named addressee(s) and may be
> confidential or proprietary in nature or covered by the provisions of
> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New
> Data Protection Code).Any use not in accord with its purpose, any
> disclosure, reproduction, copying, distribution, or either
> dissemination, either whole or partial, is strictly forbidden except
> previous formal approval of the named addressee(s). If you are not the
> intended recipient, please contact immediately the sender by
> telephone, fax or e-mail and delete the information in this message
> that has been received in error. The sender does not give any warranty
> or accept liability as the content, accuracy or completeness of sent
> messages and accepts no responsibility  for changes made after they
> were sent or for other risks which arise as a result of e-mail
> transmission, viruses, etc.
>
>
> On Wed, Apr 6, 2016 at 10:33 AM, Niels Charlier  wrote:
>> Hello Devon,
>>
>> Directory walking is part of the harvesting process, which is at the moment
>> implemented inside the ImageMosaicReader (see the inner class
>> HarvestedResource).
>> In the API, there is a separate MosaicHarvester interface, as well as a very
>> light ImageMosaicReader with all the logic inside the ImageMosaic class.
>> With this API, we can either do one of these two thing to customise
>> harvesting:
>> 1. Write our own reader and use all the mosaic functionality through the
>> ImageMosaic class.
>> 2. Write our own implementation of MosaicHarvester for a customised
>> harvesting process and plug it into the ImageMosaicReader.
>>
>> Does that make any sense?
>>
>> Kind Regards
>> Niels
>>
>>
>> On 06-04-16 01:48, Devon Tucker wrote:
>>
>> For my own part, Niels maybe you could throw in some small examples of how
>> these API changes would be used in practice to accomplish some of the items
>> in the description? For example, how could I customize the directory walking
>> with the new API?
>>
>> Thanks,
>> Devon
>>
>> On Tue, Apr 5, 2016 at 4:44 PM, Devon Tucker  wrote:
>>> Hi all,
>>>
>>> Just noticed that there wasn't a dedicated email thread for this proposal
>>> and figured I'd kick one off since I've also been involved in it.
>>>
>>>
>>> https://github.com/geotools/geotools/wiki/Refactor-ImageMosaic-for-extensibility
>>>
>>> I think the description gives a pretty good rationale for what we want to
>>> accomplish with this proposal and I'd like to solicit any discussion,
>>> advice, feedback, etc. from everyone.
>>>
>>> Cheers,
>>> Devon
>>
>>
>>
>> --
>>
>>
>>
>> ___
>> 

[Geotools-devel] memory datastore refactor

[Jody Garnett]

A small pull request here ,
refactoring MemoryDataStore to use each entry to store features (rather
than a Map