[GitHub] [sedona] jiayuasu commented on pull request #748: [SEDONA-233] Incorrect results for several joins in a single stage
jiayuasu commented on PR #748: URL: https://github.com/apache/sedona/pull/748#issuecomment-1402500468 @yitao-li Dear Yitao, Sedona R build started to fail since this PR. But the PR is not relevant to the R side, could you please take a look? Thanks, Jia -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu closed pull request #710: Bump pyspark from 3.1.2 to 3.2.2 in /binder
jiayuasu closed pull request #710: Bump pyspark from 3.1.2 to 3.2.2 in /binder URL: https://github.com/apache/sedona/pull/710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu opened a new pull request, #750: [DOCS] Remove all incubator stuff and fix dead links due to version selector
jiayuasu opened a new pull request, #750: URL: https://github.com/apache/sedona/pull/750 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/community/develop/) ## Is this PR related to a JIRA ticket? - No, this is a documentation update. The PR name follows the format `[DOCS] my subject`. ## What changes were proposed in this PR? 1. Removed all incubator stuff as Sedona has graduated to a Top Level Project. 2. The next release version is 1.4.0 3. Fixed all dead links caused by the version selector ## How was this patch tested? ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #750: [DOCS] Remove all incubator stuff and fix dead links due to version selector
jiayuasu merged PR #750: URL: https://github.com/apache/sedona/pull/750 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] umartin opened a new pull request, #751: [SEDONA-234] ST_Point inconsistencies
umartin opened a new pull request, #751: URL: https://github.com/apache/sedona/pull/751 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the assoicated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-234. The PR name follows the format `[SEDONA-XXX] my subject`. ## What changes were proposed in this PR? Drop the optional z parameter in ST_Point Add null safety to ST_Point Add ST_PointZ for 3D points ## How was this patch tested? Tests added ## Did this PR include necessary documentation updates? - Yes, I am adding a new API. I am using the [current SNAPSHOT version number](https://github.com/apache/sedona/blob/master/pom.xml#L29) in since `vX.Y.Z` format. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #751: [SEDONA-234] ST_Point inconsistencies
jiayuasu merged PR #751: URL: https://github.com/apache/sedona/pull/751 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] sebastienbourg opened a new issue, #752: ShapefileReader returning empty rawSpatialRDD when containing POLYGON Z column
sebastienbourg opened a new issue, #752:
URL: https://github.com/apache/sedona/issues/752
## Expected behavior
Read shapefile with PolygonZ
## Actual behavior
Hello,
I'm trying to read a shapefile containing POLYGON Z in geometry column. I
used Python API and also tried with Scala API.
When using ShapefileReader.readToGeometryRDD(sc = spark,
inputPath='/path/to/shapefolder/')
It returns an empty rdd : rddp.countWithoutDuplicates() return 0
Also when using the Adapter to convert it to DadtaFrame it's still empty.
The metadata are well read perhaps.
When using geopandas it works fine but it take a lot of time.
## Steps to reproduce the problem
`import os
import geopandas as gpd
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, expr, when
from sedona.register import SedonaRegistrator
from sedona.utils import SedonaKryoRegistrator, KryoSerializer
from sedona.core.formatMapper.shapefileParser import ShapefileReader
from sedona.utils.adapter import Adapter
from sedona.core.enums import GridType
from sedona.core.enums import IndexType
from sedona.core.spatialOperator import JoinQueryRaw
spark = SparkSession.builder.\
master("local[*]").\
appName("SedonaSQL-Example").\
config("spark.serializer", KryoSerializer.getName).\
config("spark.kryo.registrator", SedonaKryoRegistrator.getName) .\
config('spark.jars.packages',
'org.apache.sedona:sedona-python-adapter-3.0_2.12:1.3.1-incubating,'
'org.datasyslab:geotools-wrapper:1.1.0-25.2'). \
getOrCreate()
SedonaRegistrator.registerAll(spark)
sc = spark.sparkContext
rddp = ShapefileReader.readToGeometryRDD(sc = spark,
inputPath='/path/to/shapefolder/' )
rddp.countWithoutDuplicates()
rddp.rawSpatialRDD.count()
`
## Settings
Sedona version = 1.3.1
Apache Spark version = 3.3.0
API type = Python (Scala also)
Scala version =2.12
JRE version = 1.8
Python version = 3.9.5
Environment = Azure Databricks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on issue #752: ShapefileReader returning empty rawSpatialRDD when containing POLYGON Z column
jiayuasu commented on issue #752: URL: https://github.com/apache/sedona/issues/752#issuecomment-1410852538 @sebastienbourg Thanks for reporting this issue. According to SEDONA-163, Shapefile Reader currently only supports the types marked as `true` below. Geometries that are in other types will be automatically skipped. If you check your log, you should be able to see a number of WARNING log. For POLYGONZ shapefile, I suggest you use GeoPandas to convert it to WKT/WKB. ``` NULL(0, false), POINT(1, true), POLYLINE(3, true), POLYGON(5, true), MULTIPOINT(8, true), POINTZ(11, false), POLYLINEZ(13, false), POLYGONZ(15, false), MULTIPOINTZ(18, false), POINTM(21, false), POLYLINEM(23, false), POLYGONM(25, false), MULTIPOINTM(28, false), MULTIPATCH(31, false), ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] sebastienbourg commented on issue #752: ShapefileReader returning empty rawSpatialRDD when containing POLYGON Z column
sebastienbourg commented on issue #752: URL: https://github.com/apache/sedona/issues/752#issuecomment-1411709776 @jiayuasu Thanks a lot for your reply. We will try this way ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu closed issue #752: ShapefileReader returning empty rawSpatialRDD when containing POLYGON Z column
jiayuasu closed issue #752: ShapefileReader returning empty rawSpatialRDD when containing POLYGON Z column URL: https://github.com/apache/sedona/issues/752 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation opened a new pull request, #753: [SEDONA-236] Fix flaky python tests related to converting GeoPandas DataFrames to Spark DataFrames
Kontinuation opened a new pull request, #753: URL: https://github.com/apache/sedona/pull/753 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-236. The PR name follows the format `[SEDONA-XXX] my subject`. ## What changes were proposed in this PR? Fix test failures caused by converting pandas DataFrames containing lots of missing values to Spark DataFrames. We replaced missing values in the pandas DataFrame to get rid of this problem. ## How was this patch tested? This patch was verified on an AWS EC2 instance where the problem could be constantly reproduced. ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #753: [SEDONA-236] Fix flaky python tests related to converting GeoPandas DataFrames to Spark DataFrames
jiayuasu merged PR #753: URL: https://github.com/apache/sedona/pull/753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] YangXiaojie1998 opened a new issue, #754: Python tesT
YangXiaojie1998 opened a new issue, #754: URL: https://github.com/apache/sedona/issues/754 ## Expected behavior ## Actual behavior ## Steps to reproduce the problem ## Settings Sedona version = ? Apache Spark version = ? Apache Flink version = ? API type = Scala, Java, Python? Scala version = 2.11, 2.12, 2.13? JRE version = 1.8, 1.11? Python version = ? Environment = Standalone, AWS EC2, EMR, Azure, Databricks? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] YangXiaojie1998 closed issue #754: Python tesT
YangXiaojie1998 closed issue #754: Python tesT URL: https://github.com/apache/sedona/issues/754 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] BarbieQkiller opened a new pull request, #755: [SEDONA-179] Claim OGC compliance.
BarbieQkiller opened a new pull request, #755: URL: https://github.com/apache/sedona/pull/755 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the assoicated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-179. The PR name follows the format `[SEDONA-179] my subject`. ## What changes were proposed in this PR? The most relevant parts of OGC sfs tests for Geometry types and functions are implemented. Findings: Two functions are not compliant plus some functions are not implemented in Sedona. Tests for those are ignored with comments. ## How was this patch tested? Test suite. ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #755: [SEDONA-179] Claim OGC compliance.
jiayuasu commented on PR #755: URL: https://github.com/apache/sedona/pull/755#issuecomment-1418683926 @BarbieQkiller Thanks for this great PR. I have updated SEDONA-179 to reflect the latest progress and I also created a number of tickets to fix these tests. A few questions: 1. I didn't see Test 51 here. Is there a particular reason? 2. I didn't see PostGIS implements PolygonFromWKB for Test 9. Do you know how they handle this case? Thanks, Jia -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] BarbieQkiller commented on pull request #755: [SEDONA-179] Claim OGC compliance.
BarbieQkiller commented on PR #755: URL: https://github.com/apache/sedona/pull/755#issuecomment-1418902393 > @BarbieQkiller Thanks for this great PR. I have updated [SEDONA-179](https://issues.apache.org/jira/browse/SEDONA-179) to reflect the latest progress and I also created a number of tickets to fix these tests. > > A few questions: > > 1. I didn't see Test 51 here. Is there a particular reason? > 2. I didn't see PostGIS implements PolygonFromWKB for Test 9. Do you know how they handle this case? > > Thanks, Jia Thank you for your kind response! 1. I simply missed that one, apologies. I will update the PR. 2. T9 verifies ST_AsBinary the missing PolygonFromWKB shouldn't be a problem. https://github.com/postgis/postgis/blob/f6def67654c25d812446239036cee44812613748/extras/ogc_test_suite/2_queries.sql#L255 Best regards /Angela -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] BarbieQkiller commented on pull request #755: [SEDONA-179] Claim OGC compliance.
BarbieQkiller commented on PR #755: URL: https://github.com/apache/sedona/pull/755#issuecomment-1419125944 Added the missing test. BR, /Angela -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #755: [SEDONA-179] Claim OGC compliance
jiayuasu merged PR #755: URL: https://github.com/apache/sedona/pull/755 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Imbruced closed pull request #696: Sedona 173 delta lake tests
Imbruced closed pull request #696: Sedona 173 delta lake tests URL: https://github.com/apache/sedona/pull/696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jbampton opened a new pull request, #756: [DOCS] fix spelling
jbampton opened a new pull request, #756: URL: https://github.com/apache/sedona/pull/756 ## Did you read the Contributor Guide? - Yes, but both links in the "Yes" option are broken with 404 not found: https://sedona.apache.org/community/rule/ https://sedona.apache.org/community/develop/ I think it should be https://sedona.apache.org/latest-snapshot/community/rule/ etc ## Is this PR related to a JIRA ticket? - No, this is a documentation update. The PR name follows the format `[DOCS] my subject`. ## What changes were proposed in this PR? Fixing some typos ## How was this patch tested? ## Did this PR include necessary documentation updates? - Yes, I am adding a new API. I am using the [current SNAPSHOT version number](https://github.com/apache/sedona/blob/master/pom.xml#L29) in since `vX.Y.Z` format. - Yes, I have updated the documentation update. - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #756: [DOCS] fix spelling
jiayuasu merged PR #756: URL: https://github.com/apache/sedona/pull/756 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jbampton opened a new pull request, #757: [DOCS] Fix broken links, spelling and whitespace
jbampton opened a new pull request, #757: URL: https://github.com/apache/sedona/pull/757 Remove unneeded space from inside code span element sedona % mdl -r MD038 docs docs/tutorial/rdd.md:66: MD038 Spaces inside code span elements https://github.com/markdownlint/markdownlint https://github.com/markdownlint/markdownlint/blob/main/docs/RULES.md#md038---spaces-inside-code-span-elements ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/community/develop/) ## Is this PR related to a JIRA ticket? - No, this is a documentation update. The PR name follows the format `[DOCS] my subject`. ## What changes were proposed in this PR? Mainly fixing links and general clean up Spelling `assoicated ` -> `associated ` ## How was this patch tested? ## Did this PR include necessary documentation updates? - Yes, I am adding a new API. I am using the [current SNAPSHOT version number](https://github.com/apache/sedona/blob/master/pom.xml#L29) in since `vX.Y.Z` format. - Yes, I have updated the documentation update. - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #757: [DOCS] Fix broken links, spelling and whitespace
jiayuasu merged PR #757: URL: https://github.com/apache/sedona/pull/757 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jbampton opened a new pull request, #758: Fix spelling in Markdown and Python files
jbampton opened a new pull request, #758: URL: https://github.com/apache/sedona/pull/758 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-XXX. The PR name follows the format `[SEDONA-XXX] my subject`. - No, this is a documentation update. The PR name follows the format `[DOCS] my subject`. ## What changes were proposed in this PR? Fix typos in Markdown. One spelling fix in one Python file. ## How was this patch tested? ## Did this PR include necessary documentation updates? - Yes, I am adding a new API. I am using the [current SNAPSHOT version number](https://github.com/apache/sedona/blob/master/pom.xml#L29) in since `vX.Y.Z` format. - Yes, I have updated the documentation update. - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #758: [DOCS] Fix spelling in Markdown and Python files
jiayuasu merged PR #758: URL: https://github.com/apache/sedona/pull/758 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jbampton opened a new pull request, #759: [DOCS] Fix spelling
jbampton opened a new pull request, #759: URL: https://github.com/apache/sedona/pull/759 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - No, this is a documentation update. The PR name follows the format `[DOCS] my subject`. ## What changes were proposed in this PR? Fixing typos in "code" files ## How was this patch tested? ## Did this PR include necessary documentation updates? - Yes, I am adding a new API. I am using the [current SNAPSHOT version number](https://github.com/apache/sedona/blob/master/pom.xml#L29) in since `vX.Y.Z` format. - Yes, I have updated the documentation update. - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #759: [DOCS] Fix spelling
jiayuasu merged PR #759: URL: https://github.com/apache/sedona/pull/759 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jbampton opened a new pull request, #760: [DOCS] Fix spelling
jbampton opened a new pull request, #760: URL: https://github.com/apache/sedona/pull/760 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - No, this is a documentation update. The PR name follows the format `[DOCS] my subject`. ## What changes were proposed in this PR? Fixing typos in "code" files ## How was this patch tested? ## Did this PR include necessary documentation updates? - Yes, I am adding a new API. I am using the [current SNAPSHOT version number](https://github.com/apache/sedona/blob/master/pom.xml#L29) in since `vX.Y.Z` format. - Yes, I have updated the documentation update. - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #760: [DOCS] Fix spelling
jiayuasu merged PR #760: URL: https://github.com/apache/sedona/pull/760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jbampton opened a new pull request, #761: [DOCS] Fix spelling
jbampton opened a new pull request, #761: URL: https://github.com/apache/sedona/pull/761 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - No, this is a documentation update. The PR name follows the format `[DOCS] my subject`. ## What changes were proposed in this PR? Fixes typos in Java files ## How was this patch tested? ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jbampton opened a new issue, #762: The `Google Form` for feedback is no longer accepting responses
jbampton opened a new issue, #762: URL: https://github.com/apache/sedona/issues/762 ## Expected behavior Form is still open for feedback. ## Actual behavior "The form Feedback to Improve Apache Sedona is no longer accepting responses." ## Steps to reproduce the problem On the website open the Google Form link: https://sedona.apache.org/latest-snapshot/community/contact/#feedback -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jbampton commented on a diff in pull request #662: [SEDONA-142] Add ST_Collect to Flink Catalog
jbampton commented on code in PR #662:
URL: https://github.com/apache/sedona/pull/662#discussion_r1103674823
##
flink/src/main/java/org/apache/sedona/flink/SedonaFlinkRegistrator.java:
##
@@ -36,9 +41,24 @@ public static void registerFunc(StreamTableEnvironment
tblEnv) {
);
}
+/**
+ * This is needed because for Resolving the RAW Geometry DataType in
ST_Collect (MapView in Accumulator)
+ * the serializer needs to be known. To get the registered Serializer the
ExecutionEnvironment is needed and
+ * to add the UDF the TableEnvironment ist needed.
Review Comment:
```suggestion
* to add the UDF the TableEnvironment is needed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] jbampton opened a new pull request, #763: [DOCS] Markdown: Standardize code block linguist languages
jbampton opened a new pull request, #763: URL: https://github.com/apache/sedona/pull/763 Remove unneeded whitespace and use lowercase https://github.com/github/linguist/blob/master/lib/linguist/languages.yml Remove whitespace at end of code blocks ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - No, this is a documentation update. The PR name follows the format `[DOCS] my subject`. ## What changes were proposed in this PR? Standardizing the Markdown code blocks ## How was this patch tested? ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] zongsizhang opened a new pull request, #764: [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang opened a new pull request, #764: URL: https://github.com/apache/sedona/pull/764 …unction ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) - No, I haven't read it. ## Is this PR related to a JIRA ticket? - Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-XXX. The PR name follows the format `[SEDONA-XXX] my subject`. - No, this is a documentation update. The PR name follows the format `[DOCS] my subject`. ## What changes were proposed in this PR? Add integration with Google S2 into Common https://s2geometry.io/ Add function to cover a geometry with s2Cells and return the corresponding cellIds. Add spark core and sql functions to generate S2Cells from gemetry. ## How was this patch tested? ## Did this PR include necessary documentation updates? - Yes, I am adding a new API. I am using the [current SNAPSHOT version number] - Yes, I have updated the documentation update. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kimahriman commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
Kimahriman commented on code in PR #764:
URL: https://github.com/apache/sedona/pull/764#discussion_r1103834778
##
sql/pom.xml:
##
@@ -123,6 +123,11 @@
org.scalatest
scalatest_${scala.compat.version}
+
+com.google.geometry
+s2-geometry
+2.0.0
+
Review Comment:
This is never reference directly in this module so I don't think you need to
include this dependency here
##
common/pom.xml:
##
@@ -57,6 +57,11 @@
org.wololo
jts2geojson
+
+com.google.geometry
+s2-geometry
+2.0.0
+
Review Comment:
Add this to the dependencyManagement section of the parent pom and only
specify the version there
##
common/src/main/java/org/apache/sedona/common/utils/GeomUtils.java:
##
@@ -27,6 +27,8 @@
import java.nio.ByteOrder;
import java.util.*;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.LinkedBlockingQueue;
Review Comment:
Doesn't look like these are used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #761: [DOCS] Fix spelling
jiayuasu merged PR #761: URL: https://github.com/apache/sedona/pull/761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #763: [DOCS] Markdown: Standardize code block linguist languages
jiayuasu merged PR #763: URL: https://github.com/apache/sedona/pull/763 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
jiayuasu commented on code in PR #764:
URL: https://github.com/apache/sedona/pull/764#discussion_r1103891926
##
common/src/test/java/org/apache/sedona/common/FunctionsTest.java:
##
@@ -238,4 +235,98 @@ public void splitHeterogeneousGeometryCollection() {
assertNull(actualResult);
}
+
+@Test
Review Comment:
Please add more tests for different types (7 types in total)
##
sql/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/Functions.scala:
##
@@ -1055,3 +1055,11 @@ case class ST_Split(inputExpressions: Seq[Expression])
copy(inputExpressions = newChildren)
}
}
+
Review Comment:
Please also add this SQL function to Sedona Flink module
##
common/src/main/java/org/apache/sedona/common/Functions.java:
##
@@ -543,4 +542,25 @@ public static Geometry split(Geometry input, Geometry
blade) {
// check input geometry
return new GeometrySplitter(GEOMETRY_FACTORY).split(input, blade);
}
+
+
+/**
+ * get the coordinates of a geometry and transform to Google s2 cell id
+ * @param input Geometry
+ * @param level integer, minimum level of cells covering the geometry
+ * @return List of coordinates
+ */
+public static Long[] s2CellIDs(Geometry input, int level) {
+HashSet cellIds = new HashSet<>();
+List geoms = GeomUtils.extractGeometryCollection(input);
+for (Geometry geom : geoms) {
+try {
+
cellIds.addAll(S2Utils.s2RegionToCellIDs(S2Utils.toS2Region(geom), 1, level,
Integer.MAX_VALUE - 1));
Review Comment:
More case handlers are needed. Currently, Sedona supports 7 most common
geometry types
instanceOf Point
instanceOf MuliPoint
instanceOf Polygon
instanceOf MultiPolygon
instanceOf LineString
instanceOf MultiLineString
instanceOf GeometryCollection
##
docs/api/sql/Function.md:
##
@@ -1536,3 +1536,20 @@ SELECT ST_ZMin(ST_GeomFromText('LINESTRING(1 3 4, 5 6
7)'))
```
Output: `4.0`
+
+## ST_S2CellIDs
+
+Introduction: Cover the geometry with Google S2 Cells, return the
corresponding cell IDs with the given level.
+The level indicates the [size of
cells](https://s2geometry.io/resources/s2cell_statistics.html). With a bigger
level,
+the cells will be smaller, the coverage will be more accurate, but the result
size will be exponentially increasing.
+
+Format: `ST_S2CellIDs(geom: geometry, level: Int)`
+
+Since: `v1.3.2`
Review Comment:
Please use version 1.4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] zongsizhang commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang commented on code in PR #764:
URL: https://github.com/apache/sedona/pull/764#discussion_r1104396437
##
common/src/main/java/org/apache/sedona/common/Functions.java:
##
@@ -543,4 +542,25 @@ public static Geometry split(Geometry input, Geometry
blade) {
// check input geometry
return new GeometrySplitter(GEOMETRY_FACTORY).split(input, blade);
}
+
+
+/**
+ * get the coordinates of a geometry and transform to Google s2 cell id
+ * @param input Geometry
+ * @param level integer, minimum level of cells covering the geometry
+ * @return List of coordinates
+ */
+public static Long[] s2CellIDs(Geometry input, int level) {
+HashSet cellIds = new HashSet<>();
+List geoms = GeomUtils.extractGeometryCollection(input);
+for (Geometry geom : geoms) {
+try {
+
cellIds.addAll(S2Utils.s2RegionToCellIDs(S2Utils.toS2Region(geom), 1, level,
Integer.MAX_VALUE - 1));
Review Comment:
they are all supported already. MultiPoint, MultiPolygon, MultiLineString
are GeometryCollection, after extraction they become points, polygons and
linestrings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] zongsizhang commented on pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang commented on PR #764: URL: https://github.com/apache/sedona/pull/764#issuecomment-1427895960 @jiayuasu do you know where is this error from? https://github.com/apache/sedona/actions/runs/4157076083/jobs/7191302226#step:18:218 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] zongsizhang commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang commented on code in PR #764:
URL: https://github.com/apache/sedona/pull/764#discussion_r1104440607
##
docs/api/sql/Function.md:
##
@@ -1536,3 +1536,20 @@ SELECT ST_ZMin(ST_GeomFromText('LINESTRING(1 3 4, 5 6
7)'))
```
Output: `4.0`
+
+## ST_S2CellIDs
+
+Introduction: Cover the geometry with Google S2 Cells, return the
corresponding cell IDs with the given level.
+The level indicates the [size of
cells](https://s2geometry.io/resources/s2cell_statistics.html). With a bigger
level,
+the cells will be smaller, the coverage will be more accurate, but the result
size will be exponentially increasing.
+
+Format: `ST_S2CellIDs(geom: geometry, level: Int)`
+
+Since: `v1.3.2`
Review Comment:
changed to 1.4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] zongsizhang commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang commented on code in PR #764:
URL: https://github.com/apache/sedona/pull/764#discussion_r1104441106
##
common/src/test/java/org/apache/sedona/common/FunctionsTest.java:
##
@@ -238,4 +235,98 @@ public void splitHeterogeneousGeometryCollection() {
assertNull(actualResult);
}
+
+@Test
Review Comment:
added test case for all 7 types
##
common/src/test/java/org/apache/sedona/common/FunctionsTest.java:
##
@@ -238,4 +235,98 @@ public void splitHeterogeneousGeometryCollection() {
assertNull(actualResult);
}
+
+@Test
Review Comment:
added test case for all 7 types, please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] zongsizhang commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang commented on code in PR #764: URL: https://github.com/apache/sedona/pull/764#discussion_r1104441429 ## common/src/main/java/org/apache/sedona/common/utils/GeomUtils.java: ## @@ -27,6 +27,8 @@ import java.nio.ByteOrder; import java.util.*; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; Review Comment: removed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] zongsizhang commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang commented on code in PR #764:
URL: https://github.com/apache/sedona/pull/764#discussion_r1104442080
##
sql/pom.xml:
##
@@ -123,6 +123,11 @@
org.scalatest
scalatest_${scala.compat.version}
+
+com.google.geometry
+s2-geometry
+2.0.0
+
Review Comment:
thanks have removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] zongsizhang commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang commented on code in PR #764: URL: https://github.com/apache/sedona/pull/764#discussion_r1104443522 ## common/pom.xml: ## @@ -57,6 +57,11 @@ org.wololo jts2geojson + +com.google.geometry +s2-geometry +2.0.0 + Review Comment: thanks, change committed, please help review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jbampton opened a new pull request, #765: [DOCS] Fix spelling
jbampton opened a new pull request, #765: URL: https://github.com/apache/sedona/pull/765 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - No, this is a documentation update. The PR name follows the format `[DOCS] my subject`. ## What changes were proposed in this PR? Fixing spelling ## How was this patch tested? ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
jiayuasu commented on code in PR #764:
URL: https://github.com/apache/sedona/pull/764#discussion_r1105030032
##
common/src/test/java/org/apache/sedona/common/FunctionsTest.java:
##
@@ -238,4 +235,153 @@ public void splitHeterogeneousGeometryCollection() {
assertNull(actualResult);
}
+
+private static boolean intersects(Set s1, Set s2) {
+Set copy = new HashSet<>(s1);
+copy.retainAll(s2);
+return !copy.isEmpty();
+}
+
+@Test
+public void getGoogleS2CellIDsPoint() {
+Point point = GEOMETRY_FACTORY.createPoint(new Coordinate(1, 2));
+Long[] cid = Functions.s2CellIDs(point, 30);
+Polygon reversedPolygon = S2Utils.toJTSPolygon(new S2CellId(cid[0]));
+// cast the cell to a rectangle, it must be able to cover the points
+assert(reversedPolygon.contains(point));
+}
+
+@Test
+public void getGoogleS2CellIDsPolygon() {
+// polygon with holes
+Polygon target = GEOMETRY_FACTORY.createPolygon(
+GEOMETRY_FACTORY.createLinearRing(coordArray(0.1, 0.1, 0.5,
0.1, 1.0, 0.3, 1.0, 1.0, 0.1, 1.0, 0.1, 0.1)),
+new LinearRing[] {
+GEOMETRY_FACTORY.createLinearRing(coordArray(0.2, 0.2,
0.5, 0.2, 0.6, 0.7, 0.2, 0.6, 0.2, 0.2))
+}
+);
+// polygon inside the hole, shouldn't intersect with the polygon
+Polygon polygonInHole = GEOMETRY_FACTORY.createPolygon(coordArray(0.3,
0.3, 0.4, 0.3, 0.3, 0.4, 0.3, 0.3));
+// mbr of the polygon that cover all
+Geometry mbr = target.getEnvelope();
+HashSet targetCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(target, 10)));
+HashSet inHoleCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(polygonInHole, 10)));
+HashSet mbrCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(mbr, 10)));
+assert mbrCells.containsAll(targetCells);
+assert !intersects(targetCells, inHoleCells);
+assert mbrCells.containsAll(targetCells);
+}
+
+@Test
+public void getGoogleS2CellIDsLineString() {
+// polygon with holes
+LineString target = GEOMETRY_FACTORY.createLineString(coordArray(0.2,
0.2, 0.3, 0.4, 0.4, 0.6));
+LineString crossLine =
GEOMETRY_FACTORY.createLineString(coordArray(0.4, 0.1, 0.1, 0.4));
+// mbr of the polygon that cover all
+Geometry mbr = target.getEnvelope();
+// cover the target polygon, and convert cells back to polygons
+HashSet targetCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(target, 15)));
+HashSet crossCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(crossLine, 15)));
+HashSet mbrCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(mbr, 15)));
+assert intersects(targetCells, crossCells);
+assert mbrCells.containsAll(targetCells);
+}
+
+@Test
+public void getGoogleS2CellIDsMultiPolygon() {
+// polygon with holes
+Polygon[] geoms = new Polygon[] {
+GEOMETRY_FACTORY.createPolygon(coordArray(0.1, 0.1, 0.5, 0.1,
0.1, 0.6, 0.1, 0.1)),
+GEOMETRY_FACTORY.createPolygon(coordArray(0.2, 0.1, 0.6, 0.3,
0.7, 0.6, 0.2, 0.5, 0.2, 0.1))
+};
+MultiPolygon target = GEOMETRY_FACTORY.createMultiPolygon(geoms);
+Geometry mbr = target.getEnvelope();
+LinearRing surroundRing =
GEOMETRY_FACTORY.createLinearRing(coordArray(0.0, 0.0, 0.8, 0.0, 0.8, 0.8, 0.0,
0.8, 0.0, 0.0));
+System.out.println(GeomUtils.getWKT(surroundRing));
+HashSet targetCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(target, 10)));
+HashSet mbrCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(mbr, 10)));
+HashSet surroundCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(surroundRing, 10)));
+assert mbrCells.containsAll(targetCells);
+assert !intersects(targetCells, surroundCells);
+}
+
+@Test
+public void getGoogleS2CellIDsMultiLineString() {
+// polygon with holes
+MultiLineString target = GEOMETRY_FACTORY.createMultiLineString(
+new LineString[] {
+GEOMETRY_FACTORY.createLineString(coordArray(0.1, 0.1,
0.2, 0.1, 0.3, 0.4, 0.5, 0.9)),
+GEOMETRY_FACTORY.createLineString(coordArray(0.5, 0.1,
0.1, 0.5, 0.3, 0.1))
+}
+);
+Geometry mbr = target.getEnvelope();
+Point outsidePoint = GEOMETRY_FACTORY.createPoint(new Coordinate(0.3,
0.7));
Review Comment:
This variable is not used.
##
sql/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/st_functions.scala:
##
@@ -271,4 +271,7 @@ object st_functions extends DataFrameAPI {
def ST_ZMin(geometry: Column): Column = wrapExpression[ST_ZMin](geometry)
def ST_ZMin(geometry: String): Column = wrapExpression[ST_ZMin](geometry)
+
+
[GitHub] [sedona] jiayuasu commented on pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
jiayuasu commented on PR #764: URL: https://github.com/apache/sedona/pull/764#issuecomment-1428720789 > @jiayuasu do you know where is this error from? [apache/sedona/actions/runs/4157076083/jobs/7191302226#step:18:218](https://github.com/apache/sedona/actions/runs/4157076083/jobs/7191302226#step:18:218) @zongsizhang Sedona R CI is currently failing due to some unknown reason on the dependencies used in Sedona R module. We need our R maintainer to investigate this issue. Please ignore it for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #765: [DOCS] Fix spelling
jiayuasu merged PR #765: URL: https://github.com/apache/sedona/pull/765 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation opened a new pull request, #766: [SEDONA-207] Fix ambiguity of empty multi-geometries and multi geometries containing only empty geometries
Kontinuation opened a new pull request, #766: URL: https://github.com/apache/sedona/pull/766 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-207. The PR name follows the format `[SEDONA-XXX] my subject`. ## What changes were proposed in this PR? This pull request fixes the handling of empty multi-geometries in the geometry serializer. In the current implementation, we omitted structure data for multi-geometries containing only empty geometries, such as `MULTILINESTRING (EMPTY, EMPTY)`. In this case, we cannot tell if it is an empty multi-geometry or multi-geometry containing only empty geometries. This patch fixes the problem by always writing structure data for multi geometries. This patch also fixes other issues with the geometry serializer: * Python geometry serializer will drop M coordinates when deserializing XYM/XYZM geometries, instead of raising an exception. * Avoid allocating a huge amounts of memory when deserializing malformed data. ## How was this patch tested? Unit tests were added to test this patch. ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation opened a new pull request, #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
Kontinuation opened a new pull request, #767: URL: https://github.com/apache/sedona/pull/767 ## Notice This patch depends on https://github.com/apache/sedona/pull/766. ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/projects/SEDONA/issues/SEDONA-227. The PR name follows the format `[SEDONA-XXX] my subject`. ## What changes were proposed in this PR? This patch provides a Python geometry serializer implemented as a native extension. The original pure python serializer was still kept around as a fallback implementation when the native extension failed to load. Please be acknowledged that the existence of native extensions will complicate the release process of apache-sedona python packages, since we have to build wheels for various CPython versions and platforms. There is a newly added Github Action for building wheels as a reference approach. For platforms not covered by prebuilt wheels, users can easily install the package using the source distribution since the newly added extension does not require any third-party libraries to build. ## How was this patch tested? ### Multi-platform Compatibility We've added a Github Action for testing the native extension on various platforms. We've also run tests on Apple M1, so it is also expected to work on various ARM64 platforms. ### Performance ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation commented on pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
Kontinuation commented on PR #767: URL: https://github.com/apache/sedona/pull/767#issuecomment-1429056842 The [Github Action for building wheels](https://github.com/apache/sedona/actions/runs/4169994106) is unable to run. Seems that Github does not allow running `cibuildwheel` on ASF projects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #766: [SEDONA-207] Fix ambiguity of empty multi-geometries and multi geometries containing only empty geometries
jiayuasu merged PR #766: URL: https://github.com/apache/sedona/pull/766 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
jiayuasu commented on PR #767: URL: https://github.com/apache/sedona/pull/767#issuecomment-1429206878 Just create https://issues.apache.org/jira/browse/INFRA-24203 Let's see if the ASF infra team will allow this GitHub action. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation opened a new pull request, #768: [SEDONA-199] Update documentation for ST_NDims as it now supports M dimension.
Kontinuation opened a new pull request, #768: URL: https://github.com/apache/sedona/pull/768 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-199. The PR name follows the format `[SEDONA-XXX] my subject`. ## What changes were proposed in this PR? `ST_NDims` could handle M dimension correctly since https://github.com/apache/sedona/pull/739, so we remove the statements about its limitations. ## How was this patch tested? Added tests to show that `ST_NDims` works on XYM/XYZM geometries. ## Did this PR include necessary documentation updates? - Yes, I have updated the documentation update. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #768: [SEDONA-199] Update documentation for ST_NDims as it now supports M dimension.
jiayuasu merged PR #768: URL: https://github.com/apache/sedona/pull/768 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
jiayuasu commented on PR #767: URL: https://github.com/apache/sedona/pull/767#issuecomment-1429408885 @douglasdennis @umartin Doug and Martin, since you were following the new serializer improvement, any comment on this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] umartin commented on pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
umartin commented on PR #767: URL: https://github.com/apache/sedona/pull/767#issuecomment-1429499282 > @douglasdennis @umartin Doug and Martin, since you were following the new serializer improvement, any comment on this PR? I haven't worked with C in 20 years, but from my understanding, everything looks good. I don't have any objections to merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] gregleleu opened a new pull request, #769: Update dependencies + fix failing test
gregleleu opened a new pull request, #769: URL: https://github.com/apache/sedona/pull/769 ## What changes were proposed in this PR? Updating dependencies in R package ## How was this patch tested? Ran R tests ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] gregleleu commented on pull request #769: R: Update dependencies + fix failing test
gregleleu commented on PR #769: URL: https://github.com/apache/sedona/pull/769#issuecomment-1430580004 The R build has failed: it's not the old error -> all the tests failed because Spark did not start. I am unable to reproduce on my computer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] douglasdennis commented on a diff in pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
douglasdennis commented on code in PR #767: URL: https://github.com/apache/sedona/pull/767#discussion_r1106539613 ## python/src/geos_c_dyn_funcs.h: ## @@ -0,0 +1,145 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* The following are function pointers to GEOS C APIs provided by + * libgeos_c. These functions must be called after a successful invocation of + * `load_geos_c_functions` */ Review Comment: I couldn't find this function. I probably missed it though. ## python/src/geos_c_dyn_funcs.h: ## @@ -0,0 +1,145 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* The following are function pointers to GEOS C APIs provided by + * libgeos_c. These functions must be called after a successful invocation of + * `load_geos_c_functions` */ + +#include "geos_c_dyn.h" Review Comment: These headers include each other. Is that due to some nuance of C? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] douglasdennis commented on pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
douglasdennis commented on PR #767: URL: https://github.com/apache/sedona/pull/767#issuecomment-1430605878 @Kontinuation Will this work out of the box with a pip install? Like, once this gets pushed to pypi, will I be able to just do `pip install apache-sedona` and I'll get the speeded up version? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation commented on a diff in pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
Kontinuation commented on code in PR #767: URL: https://github.com/apache/sedona/pull/767#discussion_r1106549976 ## python/src/geos_c_dyn_funcs.h: ## @@ -0,0 +1,145 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* The following are function pointers to GEOS C APIs provided by + * libgeos_c. These functions must be called after a successful invocation of + * `load_geos_c_functions` */ Review Comment: The function to initialize geos_c_dyn should be `load_geos_c_library` or `load_geos_c_from_handle`. I've fixed the comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation commented on a diff in pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
Kontinuation commented on code in PR #767: URL: https://github.com/apache/sedona/pull/767#discussion_r1106551313 ## python/src/geos_c_dyn_funcs.h: ## @@ -0,0 +1,145 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* The following are function pointers to GEOS C APIs provided by + * libgeos_c. These functions must be called after a successful invocation of + * `load_geos_c_functions` */ + +#include "geos_c_dyn.h" Review Comment: I removed the recursive inclusion since geos_c_dyn_funcs.h is not meant to be a self-contained header file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation commented on pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
Kontinuation commented on PR #767: URL: https://github.com/apache/sedona/pull/767#issuecomment-1430620725 > @Kontinuation Will this work out of the box with a pip install? Like, once this gets pushed to pypi, will I be able to just do `pip install apache-sedona` and I'll get the speeded up version? In most cases, yes. We will release precompiled wheels for commonly used platforms. You'll still encounter problems on platforms not having matching wheel releases, such as Windows running on ARM64. If you do not have toolchains installed on your system, `pip install` won't be able to build the extension from the source release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] douglasdennis commented on pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
douglasdennis commented on PR #767: URL: https://github.com/apache/sedona/pull/767#issuecomment-1430623702 > > @Kontinuation Will this work out of the box with a pip install? Like, once this gets pushed to pypi, will I be able to just do `pip install apache-sedona` and I'll get the speeded up version? > > In most cases, yes. We will release precompiled wheels for commonly used platforms. > > You'll still encounter problems on platforms not having matching wheel releases, such as Windows running on ARM64. If you do not have toolchains installed on your system, `pip install` won't be able to build the extension from the source release. That's awesome! I'm excited to be able to use it. Thanks for putting so much work into this, this will all help me out quite a bit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #769: R: Update dependencies + fix failing test
jiayuasu commented on PR #769: URL: https://github.com/apache/sedona/pull/769#issuecomment-1430639834 @gregleleu I also don't know what happened... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #769: R: Update dependencies + fix failing test
jiayuasu commented on PR #769: URL: https://github.com/apache/sedona/pull/769#issuecomment-1430722842 @gregleleu It is weird that the exact CI passed on my fork: https://github.com/jiayuasu/sedona/pull/1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] gregleleu commented on pull request #769: R: Update dependencies + fix failing test
gregleleu commented on PR #769: URL: https://github.com/apache/sedona/pull/769#issuecomment-1430745783 @jiayuasu The logs were showing an issue with the jts core jars (between version 1.18.0 and 1.18.1) maybe it's an issue with resolving dependencies and/or cached versions? Also, should I bump the dependencies to the 1.4.0 versions? Eg 1.19.0 for jts core -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #769: R: Update dependencies + fix failing test
jiayuasu commented on PR #769: URL: https://github.com/apache/sedona/pull/769#issuecomment-1430784137 @gregleleu Yes, please bump the dependencies version to the 1.4.0 version. Thanks for reminding me about the cache issue. The CI passed after I cleaned up the cache! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] gregleleu commented on pull request #769: [SEDONA-242] R: Update dependencies + fix failing test
gregleleu commented on PR #769:
URL: https://github.com/apache/sedona/pull/769#issuecomment-1431564332
Bumped dependencies.
Just to be clear, the R package now loads:
* org.apache.sedona:sedona-core-3.0_2.12:1.3.1-incubating
* org.apache.sedona:sedona-sql-3.0_2.12:1.3.1-incubating
* org.apache.sedona:sedona-viz-3.0_2.12:1.3.1-incubating
* org.datasyslab:geotools-wrapper:1.3.0-27.2
* edu.ucar:cdm-core:5.4.2
* org.locationtech.jts:jts-core:1.19.0
* org.wololo:jts2geojson:0.16.1
It does not handle any of the exclusions that are in the POM.
("1.3.1-incubating" are replaced by "1.4.0-SNAPHSOT" for CICD by the
overwrite mechanism included in the package through the SEDONA_JAR_FILES env
variable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #769: [SEDONA-242] R: Update dependencies + fix failing test
jiayuasu merged PR #769: URL: https://github.com/apache/sedona/pull/769 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #767: [SEDONA-227] Implemented Python geometry serializer as a native extension
jiayuasu merged PR #767: URL: https://github.com/apache/sedona/pull/767 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] gregleleu opened a new pull request, #770: [SEDONA-XXX] R features: read/write geoparquet, get names from shapefiles
gregleleu opened a new pull request, #770: URL: https://github.com/apache/sedona/pull/770 Updates to the R package: * reading/writing geoparquet files * getting names to sdf from cases where "fieldNames" exists (shapefile) ## How was this patch tested? Added/updated the relevant R tests ## Did this PR include necessary documentation updates? Updated the R documentation only -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on a diff in pull request #770: [SEDONA-XXX] R features: read/write geoparquet, get names from shapefiles
jiayuasu commented on code in PR #770:
URL: https://github.com/apache/sedona/pull/770#discussion_r1107896644
##
R/R/data_interface.R:
##
@@ -439,6 +439,52 @@ sedona_read_shapefile <- function(sc,
new_spatial_rdd(NULL)
}
+
+#' Read a geoparquet file into a Spark DataFrame.
+#' Read a geoparquet file into a Spark DataFrame. The created dataframe is
automatically registered.
+#'
+#' @param sc A \code{spark_connection}.
+#' @param location Location of the data source.
+#' @param name The name to assign to the newly generated table.
+#'
+#'
+#' @return A SpatialRDD.
+#'
+#' @examples
+#' library(sparklyr)
+#' library(apache.sedona)
+#'
+#' sc <- spark_connect(master = "spark://HOST:PORT")
+#'
+#' if (!inherits(sc, "test_connection")) {
+#' input_location <- "/dev/null" # replace it with the path to your input
file
+#' rdd <- sedona_read_geoparquet(sc, location = input_location)
+#' }
+#'
+#' @family Sedona data interface functions
+#'
+#' @export
+sedona_read_geoparquet <- function(sc,
Review Comment:
Can we add a function that takes any format string? This way, the R
interface will be able to automatically invoke any new reader we add in the
future?
The example usage could be:
Read GeoParquet
```
sdf = sedona_read(sc, "geoparquet", location, name)
```
Read GeoTiff
```
sdf = sedona_read(sc, "geotiff", location, name)
```
##
R/R/data_interface.R:
##
@@ -589,6 +635,52 @@ sedona_save_spatial_rdd <- function(x,
)
}
+
+#' Save a Spark dataframe into a geoparquet file.
+#'
+#' Export spatial from a Spark dataframe into a geoparquet file
+#'
+#' @param x A Spark dataframe object in sparklyr or a dplyr expression
+#' representing a Spark SQL query.
+#' @param output_location Location of the output file.
+#'
+#'
+#' @return NULL
+#'
+#' @examples
+#' library(sparklyr)
+#' library(apache.sedona)
+#'
+#' sc <- spark_connect(master = "spark://HOST:PORT")
+#'
+#' if (!inherits(sc, "test_connection")) {
+#' tbl <- dplyr::tbl(
+#' sc,
+#' dplyr::sql("SELECT ST_GeomFromText('POINT(-71.064544 42.28787)') AS
`pt`")
+#' )
+#' sedona_save_geoparquet(
+#' tbl %>% dplyr::mutate(id = 1),
+#' output_location = "/tmp/pts.geoparquet"
+#' )
+#' }
+#'
+#' @family Sedona data interface functions
+#'
+#' @export
+sedona_save_geoparquet <- function(x,
+ output_location) {
Review Comment:
Same here:
We could make this save function more generic:
```
sedona_save(df, "geoparqet", location)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #770: [SEDONA-XXX] R features: read/write geoparquet, get names from shapefiles
jiayuasu commented on PR #770: URL: https://github.com/apache/sedona/pull/770#issuecomment-1432309242 @gregleleu Each Sedona PR needs to be associated with a JIRA ticket. I can create a Sedona JIRA account for you. This way, you can easily contribute to Sedona R in the future. To do so, I just need your email address. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on issue #762: The `Google Form` for feedback is no longer accepting responses
jiayuasu commented on issue #762: URL: https://github.com/apache/sedona/issues/762#issuecomment-1432310624 Thanks! Will be fixing this soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] gregleleu commented on pull request #770: [SEDONA-XXX] R features: read/write geoparquet, get names from shapefiles
gregleleu commented on PR #770: URL: https://github.com/apache/sedona/pull/770#issuecomment-1432313192 I created an account, and wanted to create an issue, but JIRA is so complicated I gave up. I'll try again next time. Regarding the "generic" function, it's not really the R way, cf. sparklyr's documentation (https://spark.rstudio.com/packages/sparklyr/latest/reference/) Any idea why the checks failed? I'm using the same test as the scala version – an exception with the words "GeoParquet file does not contain valid geo metadata" – but it raised another kind of exception for spark 3.0.3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #770: [SEDONA-XXX] R features: read/write geoparquet, get names from shapefiles
jiayuasu commented on PR #770: URL: https://github.com/apache/sedona/pull/770#issuecomment-1432343518 @gregleleu Spark Analysis exception has been there for a while: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/AnalysisException.scala Not sure why it failed. In fact, we have the same Scala tests on Spark 3.0.3 and it can successfully capture the exception. @Kontinuation Hi Kristin, any idea why this AnalysisException was not found in Spark 3.0.3? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on a diff in pull request #770: [SEDONA-XXX] R features: read/write geoparquet, get names from shapefiles
jiayuasu commented on code in PR #770:
URL: https://github.com/apache/sedona/pull/770#discussion_r1107921964
##
R/R/data_interface.R:
##
@@ -439,6 +439,52 @@ sedona_read_shapefile <- function(sc,
new_spatial_rdd(NULL)
}
+
+#' Read a geoparquet file into a Spark DataFrame.
+#' Read a geoparquet file into a Spark DataFrame. The created dataframe is
automatically registered.
+#'
+#' @param sc A \code{spark_connection}.
+#' @param location Location of the data source.
+#' @param name The name to assign to the newly generated table.
+#'
+#'
+#' @return A SpatialRDD.
+#'
+#' @examples
+#' library(sparklyr)
+#' library(apache.sedona)
+#'
+#' sc <- spark_connect(master = "spark://HOST:PORT")
+#'
+#' if (!inherits(sc, "test_connection")) {
+#' input_location <- "/dev/null" # replace it with the path to your input
file
+#' rdd <- sedona_read_geoparquet(sc, location = input_location)
+#' }
+#'
+#' @family Sedona data interface functions
+#'
+#' @export
+sedona_read_geoparquet <- function(sc,
Review Comment:
Thanks for the explanation. Makes sense to me. Would you please add one more
reader/writer `sedona_read_geotiff` and `sedona_write_geotiff`?
It works the same way as the geoparquet:
https://sedona.apache.org/latest-snapshot/api/sql/Raster-loader/#geotiff-dataframe-loader.
If you think this is too complicated, we certainly can leave it to another
PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] Kontinuation commented on pull request #770: [SEDONA-XXX] R features: read/write geoparquet, get names from shapefiles
Kontinuation commented on PR #770: URL: https://github.com/apache/sedona/pull/770#issuecomment-1432355733 > @gregleleu > > Spark Analysis exception has been there for a while: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/AnalysisException.scala > > Not sure why it failed. In fact, we have the same Scala tests on Spark 3.0.3 and it can successfully capture the exception. @Kontinuation Hi Kristin, any idea why this AnalysisException was not found in Spark 3.0.3? `AnalysisException` has some ABI-breaking changes after 3.0, so the code compiled with spark 3.3 could break on spark 3.0. Can we simply replace them with `IllegalArgumentException`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #770: [SEDONA-XXX] R features: read/write geoparquet, get names from shapefiles
jiayuasu commented on PR #770: URL: https://github.com/apache/sedona/pull/770#issuecomment-1432359190 @Kontinuation Yes, let's change it to `IllegalArgumentException`. Can you make a PR to fix it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] gregleleu commented on pull request #770: [SEDONA-243] R features: read/write geoparquet, get names from shapefiles
gregleleu commented on PR #770:
URL: https://github.com/apache/sedona/pull/770#issuecomment-1432361582
Do we keep the informative message ("GeoParquet file does not contain valid
geo metadata") if we change the class of the exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] Kontinuation commented on pull request #770: [SEDONA-243] R features: read/write geoparquet, get names from shapefiles
Kontinuation commented on PR #770:
URL: https://github.com/apache/sedona/pull/770#issuecomment-1432376907
> Do we keep the informative message ("GeoParquet file does not contain
valid geo metadata") if we change the class of the exception?
Yes, the message will be kept.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] Kontinuation opened a new pull request, #771: [SEDONA-226] Replace AnalysisException with IllegalArgumentException in GeoParquet reader since AnalysisException has breaking ABI chang
Kontinuation opened a new pull request, #771: URL: https://github.com/apache/sedona/pull/771 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-226. The PR name follows the format `[SEDONA-XXX] my subject`. ## What changes were proposed in this PR? Resolve an ABI compatibility issue spotted by https://github.com/apache/sedona/pull/770. The issue we're resolving is similar to [SEDONA-224](https://issues.apache.org/jira/browse/SEDONA-224). It requires some effort to make the GeoParquet implementation compatible with Spark 3.0~3.3. ## How was this patch tested? Added a python test case to verify ABI compatibility in case of invalid user input. ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] Kontinuation commented on pull request #770: [SEDONA-243] R features: read/write geoparquet, get names from shapefiles
Kontinuation commented on PR #770: URL: https://github.com/apache/sedona/pull/770#issuecomment-1432417888 > @Kontinuation Yes, let's change it to `IllegalArgumentException`. Can you make a PR to fix it? Submitted https://github.com/apache/sedona/pull/771 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #771: [SEDONA-226] Replace AnalysisException with IllegalArgumentException in GeoParquet reader since AnalysisException has breaking ABI changes between
jiayuasu merged PR #771: URL: https://github.com/apache/sedona/pull/771 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #770: [SEDONA-243] R features: read/write geoparquet, get names from shapefiles
jiayuasu commented on PR #770: URL: https://github.com/apache/sedona/pull/770#issuecomment-1432483118 @gregleleu #771 has been merged. Please pull from the upstream. This will trigger GitHub actions of this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] gregleleu commented on pull request #770: [SEDONA-243] R features: read/write geoparquet, get names from shapefiles
gregleleu commented on PR #770: URL: https://github.com/apache/sedona/pull/770#issuecomment-1433806591 @jiayuasu Made a few changes, notably changed the name of the write function to be coherent with the rest of the sparklyr ecosystem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] gregleleu commented on a diff in pull request #770: [SEDONA-243] R features: read/write geoparquet, get names from shapefiles
gregleleu commented on code in PR #770:
URL: https://github.com/apache/sedona/pull/770#discussion_r1109092384
##
R/R/data_interface.R:
##
@@ -439,6 +439,52 @@ sedona_read_shapefile <- function(sc,
new_spatial_rdd(NULL)
}
+
+#' Read a geoparquet file into a Spark DataFrame.
+#' Read a geoparquet file into a Spark DataFrame. The created dataframe is
automatically registered.
+#'
+#' @param sc A \code{spark_connection}.
+#' @param location Location of the data source.
+#' @param name The name to assign to the newly generated table.
+#'
+#'
+#' @return A SpatialRDD.
+#'
+#' @examples
+#' library(sparklyr)
+#' library(apache.sedona)
+#'
+#' sc <- spark_connect(master = "spark://HOST:PORT")
+#'
+#' if (!inherits(sc, "test_connection")) {
+#' input_location <- "/dev/null" # replace it with the path to your input
file
+#' rdd <- sedona_read_geoparquet(sc, location = input_location)
+#' }
+#'
+#' @family Sedona data interface functions
+#'
+#' @export
+sedona_read_geoparquet <- function(sc,
Review Comment:
Regarding the Geotiff: I started working on it. Reading the geotiff works
fine.
There is one complexity given the created dataframe is a nested object.
There is a package to help with that case: `sparklyr.nested`. I'll need more
time to figure out whether we need to list it as a dependency or just suggest
its use. And if it is required for the testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] jiayuasu merged pull request #770: [SEDONA-243] R features: read/write geoparquet, get names from shapefiles
jiayuasu merged PR #770: URL: https://github.com/apache/sedona/pull/770 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] zongsizhang commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang commented on code in PR #764:
URL: https://github.com/apache/sedona/pull/764#discussion_r1109851123
##
common/src/test/java/org/apache/sedona/common/FunctionsTest.java:
##
@@ -238,4 +235,153 @@ public void splitHeterogeneousGeometryCollection() {
assertNull(actualResult);
}
+
+private static boolean intersects(Set s1, Set s2) {
+Set copy = new HashSet<>(s1);
+copy.retainAll(s2);
+return !copy.isEmpty();
+}
+
+@Test
+public void getGoogleS2CellIDsPoint() {
+Point point = GEOMETRY_FACTORY.createPoint(new Coordinate(1, 2));
+Long[] cid = Functions.s2CellIDs(point, 30);
+Polygon reversedPolygon = S2Utils.toJTSPolygon(new S2CellId(cid[0]));
+// cast the cell to a rectangle, it must be able to cover the points
+assert(reversedPolygon.contains(point));
+}
+
+@Test
+public void getGoogleS2CellIDsPolygon() {
+// polygon with holes
+Polygon target = GEOMETRY_FACTORY.createPolygon(
+GEOMETRY_FACTORY.createLinearRing(coordArray(0.1, 0.1, 0.5,
0.1, 1.0, 0.3, 1.0, 1.0, 0.1, 1.0, 0.1, 0.1)),
+new LinearRing[] {
+GEOMETRY_FACTORY.createLinearRing(coordArray(0.2, 0.2,
0.5, 0.2, 0.6, 0.7, 0.2, 0.6, 0.2, 0.2))
+}
+);
+// polygon inside the hole, shouldn't intersect with the polygon
+Polygon polygonInHole = GEOMETRY_FACTORY.createPolygon(coordArray(0.3,
0.3, 0.4, 0.3, 0.3, 0.4, 0.3, 0.3));
+// mbr of the polygon that cover all
+Geometry mbr = target.getEnvelope();
+HashSet targetCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(target, 10)));
+HashSet inHoleCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(polygonInHole, 10)));
+HashSet mbrCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(mbr, 10)));
+assert mbrCells.containsAll(targetCells);
+assert !intersects(targetCells, inHoleCells);
+assert mbrCells.containsAll(targetCells);
+}
+
+@Test
+public void getGoogleS2CellIDsLineString() {
+// polygon with holes
+LineString target = GEOMETRY_FACTORY.createLineString(coordArray(0.2,
0.2, 0.3, 0.4, 0.4, 0.6));
+LineString crossLine =
GEOMETRY_FACTORY.createLineString(coordArray(0.4, 0.1, 0.1, 0.4));
+// mbr of the polygon that cover all
+Geometry mbr = target.getEnvelope();
+// cover the target polygon, and convert cells back to polygons
+HashSet targetCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(target, 15)));
+HashSet crossCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(crossLine, 15)));
+HashSet mbrCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(mbr, 15)));
+assert intersects(targetCells, crossCells);
+assert mbrCells.containsAll(targetCells);
+}
+
+@Test
+public void getGoogleS2CellIDsMultiPolygon() {
+// polygon with holes
+Polygon[] geoms = new Polygon[] {
+GEOMETRY_FACTORY.createPolygon(coordArray(0.1, 0.1, 0.5, 0.1,
0.1, 0.6, 0.1, 0.1)),
+GEOMETRY_FACTORY.createPolygon(coordArray(0.2, 0.1, 0.6, 0.3,
0.7, 0.6, 0.2, 0.5, 0.2, 0.1))
+};
+MultiPolygon target = GEOMETRY_FACTORY.createMultiPolygon(geoms);
+Geometry mbr = target.getEnvelope();
+LinearRing surroundRing =
GEOMETRY_FACTORY.createLinearRing(coordArray(0.0, 0.0, 0.8, 0.0, 0.8, 0.8, 0.0,
0.8, 0.0, 0.0));
+System.out.println(GeomUtils.getWKT(surroundRing));
+HashSet targetCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(target, 10)));
+HashSet mbrCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(mbr, 10)));
+HashSet surroundCells = new
HashSet<>(Arrays.asList(Functions.s2CellIDs(surroundRing, 10)));
+assert mbrCells.containsAll(targetCells);
+assert !intersects(targetCells, surroundCells);
+}
+
+@Test
+public void getGoogleS2CellIDsMultiLineString() {
+// polygon with holes
+MultiLineString target = GEOMETRY_FACTORY.createMultiLineString(
+new LineString[] {
+GEOMETRY_FACTORY.createLineString(coordArray(0.1, 0.1,
0.2, 0.1, 0.3, 0.4, 0.5, 0.9)),
+GEOMETRY_FACTORY.createLineString(coordArray(0.5, 0.1,
0.1, 0.5, 0.3, 0.1))
+}
+);
+Geometry mbr = target.getEnvelope();
+Point outsidePoint = GEOMETRY_FACTORY.createPoint(new Coordinate(0.3,
0.7));
Review Comment:
outsidePoint? it's used at row 321
```Long outsideCell = Functions.s2CellIDs(outsidePoint, 10)[0];```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about thi
[GitHub] [sedona] zongsizhang commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang commented on code in PR #764:
URL: https://github.com/apache/sedona/pull/764#discussion_r1110018835
##
docs/api/sql/Function.md:
##
@@ -1536,3 +1536,20 @@ SELECT ST_ZMin(ST_GeomFromText('LINESTRING(1 3 4, 5 6
7)'))
```
Output: `4.0`
+
+## ST_S2CellIDs
Review Comment:
changed, please check
##
sql/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/st_functions.scala:
##
@@ -271,4 +271,7 @@ object st_functions extends DataFrameAPI {
def ST_ZMin(geometry: Column): Column = wrapExpression[ST_ZMin](geometry)
def ST_ZMin(geometry: String): Column = wrapExpression[ST_ZMin](geometry)
+
+ def ST_S2CellIDs(geometry: Column, level: Int): Column =
wrapExpression[ST_S2CellIDs](geometry, level)
+ def ST_S2CellIDs(geometry: String, level: Int): Column =
wrapExpression[ST_S2CellIDs](geometry, level)
Review Comment:
done, please check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [sedona] umartin opened a new pull request, #772: [SEDONA-249] Add jvm flags for running tests on Java 17
umartin opened a new pull request, #772: URL: https://github.com/apache/sedona/pull/772 ## Did you read the Contributor Guide? - Yes, I have read [Contributor Rules](https://sedona.apache.org/latest-snapshot/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest-snapshot/community/develop/) ## Is this PR related to a JIRA ticket? - Yes, the URL of the associated JIRA ticket is https://issues.apache.org/jira/browse/SEDONA-249. The PR name follows the format `[SEDONA-XXX] my subject`. ## What changes were proposed in this PR? Add jvm flags to test plugins to make it possible to run tests on Java 17. This will make life easier for contributors running Java 17 locally. This PR does not add Java 17 to the test matrix for github actions. This PR does not make the tests pass on Java 17. Currently CRSTransformationTest.testPolygonDistanceJoinWithCRSTransformation in sedona-core fails on Java 17. ## How was this patch tested? All tests pass on Java 8. ## Did this PR include necessary documentation updates? - No, this PR does not affect any public API so no need to change the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #772: [SEDONA-249] Add jvm flags for running tests on Java 17
jiayuasu commented on PR #772: URL: https://github.com/apache/sedona/pull/772#issuecomment-1435176063 @umartin Thanks for bringing this up. Interestingly, I found CRSTransformationTest.testPolygonDistanceJoinWithCRSTransformation also failed on Java 11. I haven't figured why. Any idea? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] umartin commented on pull request #772: [SEDONA-249] Add jvm flags for running tests on Java 17
umartin commented on PR #772: URL: https://github.com/apache/sedona/pull/772#issuecomment-1435226458 I think there was a patch in a point release of Java 11 that changed the string representation of floats. For me it fails on Java 11.0.17 although I remember it passing on earlier version of Java 11. It the test reads geometries from csv files they could be interpreted differently depending on java version. I remember tweaking some tests in sedona-sql when that change hit me. https://github.com/apache/sedona/blob/master/sql/src/test/scala/org/apache/sedona/sql/functionTestScala.scala#L163 The test runs a distance join in RDD with polygons. The Circle geometry is only valid for points. Maybe the test can be removed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] jiayuasu commented on pull request #772: [SEDONA-249] Add jvm flags for running tests on Java 17
jiayuasu commented on PR #772: URL: https://github.com/apache/sedona/pull/772#issuecomment-1435323043 @umartin Yes! Can you remove that flaky test? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] umartin commented on pull request #772: [SEDONA-249] Add jvm flags for running tests on Java 17
umartin commented on PR #772: URL: https://github.com/apache/sedona/pull/772#issuecomment-1435654795 @jiayuasu No problem! I added a commit removing the test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [sedona] zongsizhang commented on a diff in pull request #764: [WIP] [SEDONA-235] Integrate S2, add ST_S2CellIDs
zongsizhang commented on code in PR #764:
URL: https://github.com/apache/sedona/pull/764#discussion_r1110019049
##
sql/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/st_functions.scala:
##
@@ -271,4 +271,7 @@ object st_functions extends DataFrameAPI {
def ST_ZMin(geometry: Column): Column = wrapExpression[ST_ZMin](geometry)
def ST_ZMin(geometry: String): Column = wrapExpression[ST_ZMin](geometry)
+
+ def ST_S2CellIDs(geometry: Column, level: Int): Column =
wrapExpression[ST_S2CellIDs](geometry, level)
+ def ST_S2CellIDs(geometry: String, level: Int): Column =
wrapExpression[ST_S2CellIDs](geometry, level)
Review Comment:
done dataframe api
python pending
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
