Re: [R-sig-Geo] Raster Data Management Advice

[Alexander Ilich]

Thank you everyone for the advice. I have some things to look into.

Thanks,
Alex

From: Michael Sumner 
Sent: Sunday, October 9, 2022 5:01 AM
To: Alexander Ilich 
Cc: r-sig-geo@r-project.org 
Subject: Re: [R-sig-Geo] Raster Data Management Advice

I would set up a polygon of the bounding box (in the native projection) of each 
raster source, and use fields on those polygons to store the details of 
interest: xmin,xmax,ymin,ymax, dimension, resolution, crs, and your other 
details. Then hone in a areas of interest for different tasks to see what set 
of overlapping data you have for it.

There's a lot of fanfare about STAC, but it's really just a JSON-format with 
some of the information you could store on a simple polygon dataset ...with 
STAC as with so many formats you'd have to shoehorn your data into that more 
restrictive form (you can always spit out STAC as a side product of your own 
rich summary for less sophisticated uses).

The crux is keeping the details of the source's native projection independent 
from the representation you use to query it spatially IMO, just record what's 
there. Further, the GDAL warper app-lib (one level below the gdalwarp.exe) is 
the right tool for doing general reads, of any number of sources into one 
specific window of your choosing in any projection (you could use your dataset 
described above to limit which sources get included). You can easily see what 
you'd get by merging any number of sources together, and of course more nuanced 
situations like a sensible background with more detailed layers merged over 
that is very valuable.

Cheers, Mike



On Sat, Oct 8, 2022 at 3:35 AM Alexander Ilich 
mailto:ail...@mail.usf.edu>> wrote:
Hi, I was wondering if anyone has some advice on how to organize raster
data so that it is easily queryable by various attributes (e.g. find me all
the rasters of data type bathymetry, collected by this organization with
10m resolution or finer ). Currently we have data on a server organized
often by when/where it was collected but that can make it difficult to find
specific rasters that meet a certain criteria. I've created a table as a
csv file on github 
<https://github.com/ailich/WFS_Multibeam_Metadata<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Failich%2FWFS_Multibeam_Metadata=05%7C01%7Cailich%40usf.edu%7Cc45fb8db339c4f5e12eb08daa9d4f41f%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638009029357802723%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=YFHL%2B7tp5i2u6vFxTo5L3zgCu6auS2ttvmttPrNjR9M%3D=0>>
 where
each row is a raster and it has various column attributes describing it
(e.g. who collected it, what sonar was used, resolution, coordinate system,
etc) and a path to the filename as a temporary solution, but I think some
type of spatial database that would allow for querying and then reading
into R as terra objects, as well as into QGIS and ArcGIS as layers for
visualization would be optimal as multiple project members use these data.
Tools I've come across that seem potentially useful include PostGIS and
Geopackage, but I'm not entirely sure how to properly set them up or if
they'd suit my needs. Any advice would be greatly appreciated.

Thanks,
Alex

[[alternative HTML version deleted]]

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org<mailto:R-sig-Geo@r-project.org>
https://stat.ethz.ch/mailman/listinfo/r-sig-geo<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-geo=05%7C01%7Cailich%40usf.edu%7Cc45fb8db339c4f5e12eb08daa9d4f41f%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638009029357802723%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=z4tucvFujxVkXKkVbv90Wx372fqYdP4VQ5R4qc3ERJ8%3D=0>


--
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsum...@gmail.com<mailto:mdsum...@gmail.com>

[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the 
sender and know the content is safe.

[[alternative HTML version deleted]]

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] Raster Data Management Advice

[Michael Sumner]

I would set up a polygon of the bounding box (in the native projection) of
each raster source, and use fields on those polygons to store the details
of interest: xmin,xmax,ymin,ymax, dimension, resolution, crs, and your
other details. Then hone in a areas of interest for different tasks to see
what set of overlapping data you have for it.

There's a lot of fanfare about STAC, but it's really just a JSON-format
with some of the information you could store on a simple polygon dataset
...with STAC as with so many formats you'd have to shoehorn your data into
that more restrictive form (you can always spit out STAC as a side product
of your own rich summary for less sophisticated uses).

The crux is keeping the details of the source's native
projection independent from the representation you use to query it
spatially IMO, just record what's there. Further, the GDAL warper app-lib
(one level below the gdalwarp.exe) is the right tool for doing general
reads, of any number of sources into one specific window of your choosing
in any projection (you could use your dataset described above to limit
which sources get included). You can easily see what you'd get by merging
any number of sources together, and of course more nuanced situations like
a sensible background with more detailed layers merged over that is very
valuable.

Cheers, Mike



On Sat, Oct 8, 2022 at 3:35 AM Alexander Ilich  wrote:

> Hi, I was wondering if anyone has some advice on how to organize raster
> data so that it is easily queryable by various attributes (e.g. find me all
> the rasters of data type bathymetry, collected by this organization with
> 10m resolution or finer ). Currently we have data on a server organized
> often by when/where it was collected but that can make it difficult to find
> specific rasters that meet a certain criteria. I've created a table as a
> csv file on github 
> where
> each row is a raster and it has various column attributes describing it
> (e.g. who collected it, what sonar was used, resolution, coordinate system,
> etc) and a path to the filename as a temporary solution, but I think some
> type of spatial database that would allow for querying and then reading
> into R as terra objects, as well as into QGIS and ArcGIS as layers for
> visualization would be optimal as multiple project members use these data.
> Tools I've come across that seem potentially useful include PostGIS and
> Geopackage, but I'm not entirely sure how to properly set them up or if
> they'd suit my needs. Any advice would be greatly appreciated.
>
> Thanks,
> Alex
>
> [[alternative HTML version deleted]]
>
> ___
> R-sig-Geo mailing list
> R-sig-Geo@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>


-- 
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsum...@gmail.com

[[alternative HTML version deleted]]

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] Raster Data Management Advice

[GilbertoCamara]

Dear Edzer and Alexander

STAC is rapidly expanding to become a “de facto” standard. STAC is being 
adopted by relevant cloud data providers of Earth observation data, such as 
Microsoft Planetary Computing and Earth on AWS. Its use is gradually replacing 
the OCG catalogue specification.

The “rstac” package is able to access STAC endpoints. For examples on how to 
use the “rstac” package directly, there is a nice tutorial by Marius Appel 
(author of the “gdalcubes” package) that shows how to combine “rstac” with 
“gdalcubes”. It’s available at https://youtu.be/Xlg__2PeTXM?t=3693

For R examples of access to large cloud computing collections using STAC under 
the hood, please take a look at the following chapter of the “sits” package 
documentation: 
https://e-sensing.github.io/sitsbook/earth-observation-data-cubes.html

Hope this helps.

Best
Gilberto

Prof Dr Gilberto Camara
Senior Researcher
National Institute for Space Research (INPE), Brazil
https://gilbertocamara.org/
=




> On 7 Oct 2022, at 16:13, Edzer Pebesma  wrote:
> 
> STAC is clearly the future of catalogues for spatial data, but not everyone 
> has gotten there yet. Searching or browsing available STACs is helped by stac 
> index, https://stacindex.org/
> 
> On 07/10/2022 18:43, Zivan Karaman wrote:
>> Hi,
>> Perhaps STAC  could help you?
>> Best,
>> Zivan
>> On Fri, Oct 7, 2022 at 6:35 PM Alexander Ilich  wrote:
>>> 
>>> Hi, I was wondering if anyone has some advice on how to organize raster
>>> data so that it is easily queryable by various attributes (e.g. find me all
>>> the rasters of data type bathymetry, collected by this organization with
>>> 10m resolution or finer ). Currently we have data on a server organized
>>> often by when/where it was collected but that can make it difficult to find
>>> specific rasters that meet a certain criteria. I've created a table as a
>>> csv file on github  where
>>> each row is a raster and it has various column attributes describing it
>>> (e.g. who collected it, what sonar was used, resolution, coordinate system,
>>> etc) and a path to the filename as a temporary solution, but I think some
>>> type of spatial database that would allow for querying and then reading
>>> into R as terra objects, as well as into QGIS and ArcGIS as layers for
>>> visualization would be optimal as multiple project members use these data.
>>> Tools I've come across that seem potentially useful include PostGIS and
>>> Geopackage, but I'm not entirely sure how to properly set them up or if
>>> they'd suit my needs. Any advice would be greatly appreciated.
>>> 
>>> Thanks,
>>> Alex
>>> 
>>> [[alternative HTML version deleted]]
>>> 
>>> ___
>>> R-sig-Geo mailing list
>>> R-sig-Geo@r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>> ___
>> R-sig-Geo mailing list
>> R-sig-Geo@r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> 
> -- 
> Edzer Pebesma
> Institute for Geoinformatics
> Heisenbergstrasse 2, 48151 Muenster, Germany
> Phone: +49 251 8333081
> 
> ___
> R-sig-Geo mailing list
> R-sig-Geo@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] Raster Data Management Advice

[Edzer Pebesma]

STAC is clearly the future of catalogues for spatial data, but not 
everyone has gotten there yet. Searching or browsing available STACs is 
helped by stac index, https://stacindex.org/


On 07/10/2022 18:43, Zivan Karaman wrote:

Hi,
Perhaps STAC  could help you?
Best,
Zivan


On Fri, Oct 7, 2022 at 6:35 PM Alexander Ilich  wrote:


Hi, I was wondering if anyone has some advice on how to organize raster
data so that it is easily queryable by various attributes (e.g. find me all
the rasters of data type bathymetry, collected by this organization with
10m resolution or finer ). Currently we have data on a server organized
often by when/where it was collected but that can make it difficult to find
specific rasters that meet a certain criteria. I've created a table as a
csv file on github  where
each row is a raster and it has various column attributes describing it
(e.g. who collected it, what sonar was used, resolution, coordinate system,
etc) and a path to the filename as a temporary solution, but I think some
type of spatial database that would allow for querying and then reading
into R as terra objects, as well as into QGIS and ArcGIS as layers for
visualization would be optimal as multiple project members use these data.
Tools I've come across that seem potentially useful include PostGIS and
Geopackage, but I'm not entirely sure how to properly set them up or if
they'd suit my needs. Any advice would be greatly appreciated.

Thanks,
Alex

 [[alternative HTML version deleted]]

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo


___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo


--
Edzer Pebesma
Institute for Geoinformatics
Heisenbergstrasse 2, 48151 Muenster, Germany
Phone: +49 251 8333081

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo

Re: [R-sig-Geo] Raster Data Management Advice

[Zivan Karaman]

Hi,
Perhaps STAC  could help you?
Best,
Zivan


On Fri, Oct 7, 2022 at 6:35 PM Alexander Ilich  wrote:
>
> Hi, I was wondering if anyone has some advice on how to organize raster
> data so that it is easily queryable by various attributes (e.g. find me all
> the rasters of data type bathymetry, collected by this organization with
> 10m resolution or finer ). Currently we have data on a server organized
> often by when/where it was collected but that can make it difficult to find
> specific rasters that meet a certain criteria. I've created a table as a
> csv file on github  where
> each row is a raster and it has various column attributes describing it
> (e.g. who collected it, what sonar was used, resolution, coordinate system,
> etc) and a path to the filename as a temporary solution, but I think some
> type of spatial database that would allow for querying and then reading
> into R as terra objects, as well as into QGIS and ArcGIS as layers for
> visualization would be optimal as multiple project members use these data.
> Tools I've come across that seem potentially useful include PostGIS and
> Geopackage, but I'm not entirely sure how to properly set them up or if
> they'd suit my needs. Any advice would be greatly appreciated.
>
> Thanks,
> Alex
>
> [[alternative HTML version deleted]]
>
> ___
> R-sig-Geo mailing list
> R-sig-Geo@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo

___
R-sig-Geo mailing list
R-sig-Geo@r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo