[GitHub] [sedona] jiayuasu merged pull request #808: [SEDONA-227] Enable building python wheels for ARM64
jiayuasu merged PR #808: URL: https://github.com/apache/sedona/pull/808 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation commented on pull request #808: [SEDONA-227] Enable building python wheels for ARM64
Kontinuation commented on PR #808: URL: https://github.com/apache/sedona/pull/808#issuecomment-1482264236 > > I think this LGTM. Ready to merge? @Kontinuation > > Just a minute. Let me try it out on AWS EC2 Graviton. A shakedown run shows that it works fine. Let's merge it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation commented on pull request #808: [SEDONA-227] Enable building python wheels for ARM64
Kontinuation commented on PR #808: URL: https://github.com/apache/sedona/pull/808#issuecomment-1482261416 > I think this LGTM. Ready to merge? @Kontinuation Just a minute. Let me try it out on AWS EC2 Graviton. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #808: [SEDONA-227] Enable building python wheels for ARM64
jiayuasu commented on PR #808: URL: https://github.com/apache/sedona/pull/808#issuecomment-1482258667 I think this LGTM. Ready to merge? @Kontinuation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation opened a new pull request, #808: [SEDONA-227] Enable building python wheels for ARM64
Kontinuation opened a new pull request, #808: URL: https://github.com/apache/sedona/pull/808 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-227. The PR name follows the format `[SEDONA-XXX] my subject`. ## What changes were proposed in this PR? Build python wheels for ARM64. ## How was this patch tested? Tested manually by installing the wheels on Linux aarch64 and macOS arm64. Windows ARM64 was left untested. ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] umartin commented on issue #807: Writing dataframe to Postgresql (Postgis-Extension) DB fails
umartin commented on issue #807: URL: https://github.com/apache/sedona/issues/807#issuecomment-1481405797 Hi, If you create the table externally to Spark before writing you can set the column type to geometry and Postgis will accept the EWKB as geometry. If you let Spark create the table you can change the column type post insert. Execute something like this in Postgis: "alter table my_table alter column geometry type geometry;". Let me know if that works for you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] conect commented on issue #807: Writing dataframe to Postgresql (Postgis-Extension) DB fails
conect commented on issue #807:
URL: https://github.com/apache/sedona/issues/807#issuecomment-1481393363
Hi @umartin,
Thank you for your fast reply!
Hm when inserting it your way (slightly java adapted)
`Dataset d2 = Adapter.toDf(spatialRDD1,spark).withColumn("geometry",
functions.expr("ST_AsEWKT(geometry)"));`
The type within the postgres db gets:
Column | Type | Collation | Nullable | Default | Storage | Stats target
| Description
--+---+---+--+-+--+--+-
geometry | **bytea** | | | | extended |
|
id | text | | | | extended |
|
some_prop| text | | | | extended |
|
but bytea is not a geometric representation within postgres.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [jira] [Created] (SEDONA-268) Add support for raster types in Geotiff writer
Sorry, that should be https://issues.apache.org/jira/browse/SEDONA-269
Br,
Martin Andersson
Den tors 23 mars 2023 kl 15:33 skrev Martin Andersson <
u.martin.anders...@gmail.com>:
> Hi Mo,
>
> This issue is only about adding support for the raster UDT in the current
> geotiff writer.
>
> I also proposed https://issues.apache.org/jira/browse/SEDONA-268 which
> could be used to implement support for many different kinds of raster
> formats.
>
> Br
> Martin Andersson
>
> Den tors 23 mars 2023 kl 14:57 skrev Mo Sarwat :
>
>> Hi Martin,
>>
>> Thanks for proposing this feature. Will the writer generate
>> cloud-optimized
>> GeoTiFF. I am asking because Cogs are becoming industry standards for
>> storing raster in object stores.
>>
>> -Mo
>>
>> On Thu, Mar 23, 2023 at 6:53 AM Martin Andersson (Jira)
>> wrote:
>>
>> > Martin Andersson created SEDONA-268:
>> > ---
>> >
>> > Summary: Add support for raster types in Geotiff writer
>> > Key: SEDONA-268
>> > URL: https://issues.apache.org/jira/browse/SEDONA-268
>> > Project: Apache Sedona
>> > Issue Type: Improvement
>> > Reporter: Martin Andersson
>> >
>> >
>> > As discussed in: [
>> > https://lists.apache.org/thread/kbwqnj7kn9omtpsoyzbn0zsslvd8cm5t]
>> >
>> > With the introduction of raster types in SEDONA-251, it is now possible
>> to
>> > enhance the existing Geotiff writer to directly support writing Geotiffs
>> > from rasters.
>> >
>> > Currently, the Geotiff writer requires six columns to create Geotiff
>> > files: origin, geometry, width, height, nBands, and data. However, for
>> > rasters, only two columns would be necessary: origin and raster.
>> >
>> > To achieve this, we propose modifying the Geotiff writer to first
>> attempt
>> > to use the existing six-column format. If the input DataFrame does not
>> meet
>> > the necessary criteria, the writer should instead look for the origin
>> > column and a column with the raster UDT.
>> >
>> > For the origin column we could use the existing configuration parameter
>> > (fieldOrigin). The raster column could be detected by type. If there are
>> > several columns of type raster, the writer would throw an exception. For
>> > DataFrames with several raster columns users would need to select one
>> at a
>> > time. Example:
>> > {code:java}
>> > df_many_rasters = ...
>> > df_many_rasters.select("origin","raster1").write...
>> > df_many_rasters.select("origin","raster2").write...
>> > {code}
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > This message was sent by Atlassian Jira
>> > (v8.20.10#820010)
>> >
>>
>
Re: [jira] [Created] (SEDONA-268) Add support for raster types in Geotiff writer
Hi Mo,
This issue is only about adding support for the raster UDT in the current
geotiff writer.
I also proposed https://issues.apache.org/jira/browse/SEDONA-268 which
could be used to implement support for many different kinds of raster
formats.
Br
Martin Andersson
Den tors 23 mars 2023 kl 14:57 skrev Mo Sarwat :
> Hi Martin,
>
> Thanks for proposing this feature. Will the writer generate cloud-optimized
> GeoTiFF. I am asking because Cogs are becoming industry standards for
> storing raster in object stores.
>
> -Mo
>
> On Thu, Mar 23, 2023 at 6:53 AM Martin Andersson (Jira)
> wrote:
>
> > Martin Andersson created SEDONA-268:
> > ---
> >
> > Summary: Add support for raster types in Geotiff writer
> > Key: SEDONA-268
> > URL: https://issues.apache.org/jira/browse/SEDONA-268
> > Project: Apache Sedona
> > Issue Type: Improvement
> > Reporter: Martin Andersson
> >
> >
> > As discussed in: [
> > https://lists.apache.org/thread/kbwqnj7kn9omtpsoyzbn0zsslvd8cm5t]
> >
> > With the introduction of raster types in SEDONA-251, it is now possible
> to
> > enhance the existing Geotiff writer to directly support writing Geotiffs
> > from rasters.
> >
> > Currently, the Geotiff writer requires six columns to create Geotiff
> > files: origin, geometry, width, height, nBands, and data. However, for
> > rasters, only two columns would be necessary: origin and raster.
> >
> > To achieve this, we propose modifying the Geotiff writer to first attempt
> > to use the existing six-column format. If the input DataFrame does not
> meet
> > the necessary criteria, the writer should instead look for the origin
> > column and a column with the raster UDT.
> >
> > For the origin column we could use the existing configuration parameter
> > (fieldOrigin). The raster column could be detected by type. If there are
> > several columns of type raster, the writer would throw an exception. For
> > DataFrames with several raster columns users would need to select one at
> a
> > time. Example:
> > {code:java}
> > df_many_rasters = ...
> > df_many_rasters.select("origin","raster1").write...
> > df_many_rasters.select("origin","raster2").write...
> > {code}
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.20.10#820010)
> >
>
[jira] [Commented] (SEDONA-269) Add data source for writing binary files
[
https://issues.apache.org/jira/browse/SEDONA-269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17704171#comment-17704171
]
Martin Andersson commented on SEDONA-269:
-
For reference, the following (quite extensive) list of raster formats are
supported by GDAL: https://gdal.org/drivers/raster/index.html
> Add data source for writing binary files
>
>
> Key: SEDONA-269
> URL: https://issues.apache.org/jira/browse/SEDONA-269
> Project: Apache Sedona
> Issue Type: Improvement
>Reporter: Martin Andersson
>Priority: Major
>
> The main use case in Sedona would be to write raster files in other formats
> than Geotiff.
> The binary file data source in Spark makes it easy to support reading
> different raster formats in Sedona. We only need to implement a function to
> convert the binary content to a raster.
> Spark doesn't offer a binary data source that supports writing. That could be
> implemented in Sedona.
> The benefits to Sedona would be:
> * Sedona would be able to support writing different raster formats by
> implementing different RS_As[ImageFormat] functions instead of creating a
> custom data source for each format. Like
> [https://postgis.net/docs/RT_reference.html#Raster_Outputs]
> * Users would be able to transport rasters in common formats to other data
> stores like RDBMS.
> Example:
> {code}
> // Writing rasters to file using the proposed data source:
> df.selectExpr("filename",
> "RS_AsPNG(raster)").write.format("binary").path()
> // Transfering rasters to Postgis:
> df.selectExpr("filename", "RS_AsWKB(raster)").write.format("jdbc").option(...)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (SEDONA-269) Add data source for writing binary files
Martin Andersson created SEDONA-269:
---
Summary: Add data source for writing binary files
Key: SEDONA-269
URL: https://issues.apache.org/jira/browse/SEDONA-269
Project: Apache Sedona
Issue Type: Improvement
Reporter: Martin Andersson
The main use case in Sedona would be to write raster files in other formats
than Geotiff.
The binary file data source in Spark makes it easy to support reading different
raster formats in Sedona. We only need to implement a function to convert the
binary content to a raster.
Spark doesn't offer a binary data source that supports writing. That could be
implemented in Sedona.
The benefits to Sedona would be:
* Sedona would be able to support writing different raster formats by
implementing different RS_As[ImageFormat] functions instead of creating a
custom data source for each format. Like
[https://postgis.net/docs/RT_reference.html#Raster_Outputs]
* Users would be able to transport rasters in common formats to other data
stores like RDBMS.
Example:
{code}
// Writing rasters to file using the proposed data source:
df.selectExpr("filename", "RS_AsPNG(raster)").write.format("binary").path()
// Transfering rasters to Postgis:
df.selectExpr("filename", "RS_AsWKB(raster)").write.format("jdbc").option(...)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [jira] [Created] (SEDONA-268) Add support for raster types in Geotiff writer
Hi Martin,
Thanks for proposing this feature. Will the writer generate cloud-optimized
GeoTiFF. I am asking because Cogs are becoming industry standards for
storing raster in object stores.
-Mo
On Thu, Mar 23, 2023 at 6:53 AM Martin Andersson (Jira)
wrote:
> Martin Andersson created SEDONA-268:
> ---
>
> Summary: Add support for raster types in Geotiff writer
> Key: SEDONA-268
> URL: https://issues.apache.org/jira/browse/SEDONA-268
> Project: Apache Sedona
> Issue Type: Improvement
> Reporter: Martin Andersson
>
>
> As discussed in: [
> https://lists.apache.org/thread/kbwqnj7kn9omtpsoyzbn0zsslvd8cm5t]
>
> With the introduction of raster types in SEDONA-251, it is now possible to
> enhance the existing Geotiff writer to directly support writing Geotiffs
> from rasters.
>
> Currently, the Geotiff writer requires six columns to create Geotiff
> files: origin, geometry, width, height, nBands, and data. However, for
> rasters, only two columns would be necessary: origin and raster.
>
> To achieve this, we propose modifying the Geotiff writer to first attempt
> to use the existing six-column format. If the input DataFrame does not meet
> the necessary criteria, the writer should instead look for the origin
> column and a column with the raster UDT.
>
> For the origin column we could use the existing configuration parameter
> (fieldOrigin). The raster column could be detected by type. If there are
> several columns of type raster, the writer would throw an exception. For
> DataFrames with several raster columns users would need to select one at a
> time. Example:
> {code:java}
> df_many_rasters = ...
> df_many_rasters.select("origin","raster1").write...
> df_many_rasters.select("origin","raster2").write...
> {code}
>
>
>
>
>
>
>
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.20.10#820010)
>
[jira] [Created] (SEDONA-268) Add support for raster types in Geotiff writer
Martin Andersson created SEDONA-268:
---
Summary: Add support for raster types in Geotiff writer
Key: SEDONA-268
URL: https://issues.apache.org/jira/browse/SEDONA-268
Project: Apache Sedona
Issue Type: Improvement
Reporter: Martin Andersson
As discussed in:
[https://lists.apache.org/thread/kbwqnj7kn9omtpsoyzbn0zsslvd8cm5t]
With the introduction of raster types in SEDONA-251, it is now possible to
enhance the existing Geotiff writer to directly support writing Geotiffs from
rasters.
Currently, the Geotiff writer requires six columns to create Geotiff files:
origin, geometry, width, height, nBands, and data. However, for rasters, only
two columns would be necessary: origin and raster.
To achieve this, we propose modifying the Geotiff writer to first attempt to
use the existing six-column format. If the input DataFrame does not meet the
necessary criteria, the writer should instead look for the origin column and a
column with the raster UDT.
For the origin column we could use the existing configuration parameter
(fieldOrigin). The raster column could be detected by type. If there are
several columns of type raster, the writer would throw an exception. For
DataFrames with several raster columns users would need to select one at a
time. Example:
{code:java}
df_many_rasters = ...
df_many_rasters.select("origin","raster1").write...
df_many_rasters.select("origin","raster2").write...
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
