[GitHub] [sedona] umartin commented on pull request #773: [SEDONA-251] Add raster type to Sedona
umartin commented on PR #773: URL: https://github.com/apache/sedona/pull/773#issuecomment-1441614714 Yes, you're right, geotools-wrapper should be updated too. I also noticed that Sedona and geotools-wrapper are on different geotools versions. I ran tests locally with geotools 28.2 and they all passed. Maybe should upgrade both Sedona and geotools-wrapper to 28.2? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #773: [SEDONA-251] Add raster type to Sedona
jiayuasu merged PR #773: URL: https://github.com/apache/sedona/pull/773 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #773: [SEDONA-251] Add raster type to Sedona
jiayuasu commented on PR #773: URL: https://github.com/apache/sedona/pull/773#issuecomment-1441493133 @umartin Thanks, Martin. I will merge this PR and let's start the discussion in the mailing list :-) BTW, maybe I should add the new geotools dependency to https://github.com/jiayuasu/geotools-wrapper? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] umartin commented on pull request #773: [SEDONA-251] Add raster type to Sedona
umartin commented on PR #773: URL: https://github.com/apache/sedona/pull/773#issuecomment-1440788786 Thanks for the feedback @kanchanchy! I don’t see a need to break any of the existing api:s. Where it makes sense, some of the current RS functions could be extended to support both raster and arrays of double. Just like ST_GeomFromWKB supports both binary and string input. The way I see it is that the new raster type would only add new functionality, not remove or break any existing api. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] kanchanchy commented on pull request #773: [SEDONA-251] Add raster type to Sedona
kanchanchy commented on PR #773: URL: https://github.com/apache/sedona/pull/773#issuecomment-1440581852 > Hi @jiayuasu, some of the important functions used by GeoTorchAI are listed below: RS_GetBand RS_Normalize RS_NormalizedDifference RS_Append Although the above-mentioned functions are the most important, GeoTorchAI also uses all other RS_ functions supported by Apache Sedona currently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] umartin commented on pull request #773: [SEDONA-251] Add raster type to Sedona
umartin commented on PR #773: URL: https://github.com/apache/sedona/pull/773#issuecomment-1440570307 Thank you @jiayuasu ! 1. Yes! Now we should be able to implement all the raster functions in Postgis. 2. I think we can adapt the GeoTiff writer to work with rasters as well as the current double arrays. We just need to figure out exactly how that would work from a user perspective. I'll start a discussion on the mailing list about that. 3. I think that the raster functions in this PR and the current ones, that operate on bands, are complementary. We just need to implement a few function bridging the two. If we can extract bands from rasters and also create rasters from bands the old and new functions will work well together. There are many examples in Postgis we could use as inspiration: https://postgis.net/docs/RT_ST_AsRaster.html https://postgis.net/docs/RT_ST_AddBand.html https://postgis.net/docs/RT_ST_DumpValues.html https://postgis.net/docs/RT_ST_DumpValues.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #773: [SEDONA-251] Add raster type to Sedona
jiayuasu commented on PR #773: URL: https://github.com/apache/sedona/pull/773#issuecomment-1439502825 @umartin Martin, thanks for this great contribution! Your insights are always helpful to this community! I am Ok with this major change to Sedona Raster module. I think this will make everything easier in Sedona raster. A few more questions: 1. With this new raster implementation, is it possible that we can easily create similar PostGIS raster functions in Sedona? (https://postgis.net/docs/RT_reference.html) 2. Is it possible that we can write GeoTiff DataFrame back to disk, e.g., RS_AsTiff(https://postgis.net/docs/RT_ST_AsTIFF.html)? The current Sedona Raster implementation can write raster back to GeoTiff: https://sedona.apache.org/latest-snapshot/api/sql/Raster-loader/#geotiff-dataframe-writer 3. Is it possible that in the future PRs, we move the existing RS function implementation to this new Raster UDT? Or do you think we should just leave them as is? @kanchanchy Kanchan, downstream projects such as GeoTorchAI (https://github.com/DataSystemsLab/GeoTorchAI) heavily depends on Sedona raster funcs. What are the most important RS functions that are used by GeoTorchAI? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] umartin opened a new pull request, #773: [SEDONA-251] Add raster type to Sedona
umartin opened a new pull request, #773: URL: https://github.com/apache/sedona/pull/773 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-251. The PR name follows the format `[SEDONA-XXX] my subject`. ## What changes were proposed in this PR? This PR adds a raster type to Sedona and some basic functions using the raster type. ## How was this patch tested? Unit tests added ## Did this PR include necessary documentation updates? - Yes, I am adding a new API. I am using the [current SNAPSHOT version number] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (SEDONA-251) Add raster type to Sedona
Martin Andersson created SEDONA-251: --- Summary: Add raster type to Sedona Key: SEDONA-251 URL: https://issues.apache.org/jira/browse/SEDONA-251 Project: Apache Sedona Issue Type: Improvement Reporter: Martin Andersson Adding a raster type would enable a lot of use cases. A raster has grid values, an evelope, a coorinate referens system and other metadata. The current raster functions operate on arrays of doubles. The metadata is kept in separate columns created by the custom data source for geotiff. That is making rasters unnecessary hard to work with. A raster type would close the gap between Sedona and other raster SQL implementations like Postgis. See discussion on Sedona mailing list: [https://lists.apache.org/thread/qdfcvxl6z5pb7m7ky5zsksyytyxqwv8c] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Add raster type to Sedona
Thanks for the link. I like how they implemented it using a custom metadata class and java.awk.image.Raster. That way the implementation isn't tied to a specific library like geotools. Some more notes in rasters: * Geotools is the most used library for rasters. Geotools license isn't compatible with Apache. I came across https://sis.apache.org/. It's an apache library for vector and raster geometries. I haven't tried it but hopefully we could use it for raster geometries. Maybe we can even drop Geotools entirely and use Apache SiS for ST_Transform, shapefiles and other stuff where Geotools is used in Sedona today. * Postgis overloads functions for both vector and raster geometries. Like ST_AsBinary(geometry: g1) and ST_AsBinary(raster: r1). Spark doesn't support overloading of functions. I think the current solution with a RS prefix for raster functions in Sedona is fine. That way we could have ST_AsBinary for vector geometries and RS_AsBinary for raster geometries. * There doesn't seem to be a widely accepted standard for raster processing in SQL. I think we would have to look at Postgis and other implementations for guidance. /Martin Den mån 21 nov. 2022 kl 09:34 skrev Jia Yu : > Hi Martin, > > I think the idea is good. I also didn't find an official Java based WKB > raster implementation. But this thread might be helpful: > > https://gis.stackexchange.com/questions/184094/how-to-insert-raster-in-postgis-from-java-jdbc > > This guy implemented some prototypes... > > Thanks, > Jia > > On Fri, Nov 18, 2022 at 8:26 AM Martin Andersson < > u.martin.anders...@gmail.com> wrote: > > > Hi all, > > > > I have a use case which involves raster geometries. Naturally I would > like > > to use Sedona for that. > > > > The current raster functions operate on arrays of doubles. That means > that > > all the spatial metadata is missing. That makes it impossible to > implement > > anything like what Postgis has: > https://postgis.net/docs/RT_reference.html > > > > Rasters have a position, srid, scale and other attributes that are needed > > for a lot of functions. > > > > If Sedona had a raster type we would have a foundation to build on. Then > > the community could add their favorite raster functions, just like with > > vector functions. > > > > The Sedona raster type could be serialized to "raster WKB" format. It's a > > format invented by Postgis and not an official standard. It just borrows > > the popular name of the proper WKB which is only for vector geometries. > > > > > https://git.osgeo.org/gitea/postgis/postgis/src/branch/master/raster/doc/RFC2-WellKnownBinaryFormat > > > > There are several independent implementations of raster WKB in C, Rust, > > Python etc. But I haven't found a JVM implementation. We could implement > a > > WKBGridCoverageReader and WKBGridCoverageWriter (implementing > > GridCoverageReader and GridCoverageWriter in geotools) and use > GridCoverage > > in our functions. I haven't used raster geometries that much. There might > > be some better options. > > > > What are your thoughts? > > > > //Martin > > > -- Hälsningar, Martin
Re: Add raster type to Sedona
Hi Martin, I think the idea is good. I also didn't find an official Java based WKB raster implementation. But this thread might be helpful: https://gis.stackexchange.com/questions/184094/how-to-insert-raster-in-postgis-from-java-jdbc This guy implemented some prototypes... Thanks, Jia On Fri, Nov 18, 2022 at 8:26 AM Martin Andersson < u.martin.anders...@gmail.com> wrote: > Hi all, > > I have a use case which involves raster geometries. Naturally I would like > to use Sedona for that. > > The current raster functions operate on arrays of doubles. That means that > all the spatial metadata is missing. That makes it impossible to implement > anything like what Postgis has: https://postgis.net/docs/RT_reference.html > > Rasters have a position, srid, scale and other attributes that are needed > for a lot of functions. > > If Sedona had a raster type we would have a foundation to build on. Then > the community could add their favorite raster functions, just like with > vector functions. > > The Sedona raster type could be serialized to "raster WKB" format. It's a > format invented by Postgis and not an official standard. It just borrows > the popular name of the proper WKB which is only for vector geometries. > > https://git.osgeo.org/gitea/postgis/postgis/src/branch/master/raster/doc/RFC2-WellKnownBinaryFormat > > There are several independent implementations of raster WKB in C, Rust, > Python etc. But I haven't found a JVM implementation. We could implement a > WKBGridCoverageReader and WKBGridCoverageWriter (implementing > GridCoverageReader and GridCoverageWriter in geotools) and use GridCoverage > in our functions. I haven't used raster geometries that much. There might > be some better options. > > What are your thoughts? > > //Martin >
Add raster type to Sedona
Hi all, I have a use case which involves raster geometries. Naturally I would like to use Sedona for that. The current raster functions operate on arrays of doubles. That means that all the spatial metadata is missing. That makes it impossible to implement anything like what Postgis has: https://postgis.net/docs/RT_reference.html Rasters have a position, srid, scale and other attributes that are needed for a lot of functions. If Sedona had a raster type we would have a foundation to build on. Then the community could add their favorite raster functions, just like with vector functions. The Sedona raster type could be serialized to "raster WKB" format. It's a format invented by Postgis and not an official standard. It just borrows the popular name of the proper WKB which is only for vector geometries. https://git.osgeo.org/gitea/postgis/postgis/src/branch/master/raster/doc/RFC2-WellKnownBinaryFormat There are several independent implementations of raster WKB in C, Rust, Python etc. But I haven't found a JVM implementation. We could implement a WKBGridCoverageReader and WKBGridCoverageWriter (implementing GridCoverageReader and GridCoverageWriter in geotools) and use GridCoverage in our functions. I haven't used raster geometries that much. There might be some better options. What are your thoughts? //Martin
