Re: [DISCUSS] Graduate Apache Sedona to a Top Level Project
+1 really great project :) On Sat, Dec 3, 2022, 20:27 George Percivall wrote: > +1 > > Apache Sedona’s graduation will be a great addition to the geospatial > capabilities of Apache TLPs > > > George Percivall > perciv...@apache.org > > > > > On Dec 3, 2022, at 3:07 AM, Bertil Chapuis wrote: > > > > +1 (non-binding) > > > > Sedona is an awesome project! > > > > Best, > > > > Bertil > > > >> On 3 Dec 2022, at 01:06, PJ Fanning wrote: > >> > >> +1 (binding) - Sedona project seems to be in good shape and the team > >> looks ready to become a TLP > >> > >> On Sat, 3 Dec 2022 at 00:21, Sunil Govindan wrote: > >>> > >>> +1 (binding) > >>> > >>> Thanks for shepherding this. All the best. > >>> > >>> Thanks > >>> Sunil > >>> > >>> On Fri, Dec 2, 2022 at 6:56 AM Calvin Kirs wrote: > >>> > >>>> Hi, > >>>> > >>>> +1, I think it's time to graduate. > >>>> > >>>> On Fri, Dec 2, 2022 at 5:28 PM Jia Yu wrote: > >>>>> > >>>>> Dear Apache Incubator Community (CCed Sedona dev), > >>>>> > >>>>> Apache Sedona has been incubating since 07-19-2020. For over 2 > years, the > >>>>> Sedona community has made great progress and grown rapidly under the > >>>>> guidance of our mentors. We believe the Sedona project has met the > >>>>> conditions for graduation. We'd like to start the discussion and > collect > >>>>> feedback towards graduation. Please feel free to join the discussion. > >>>>> > >>>>> The discussion will remain open for at least 72 hours. > >>>>> > >>>>> Thanks, > >>>>> Jia > >>>>> > >>>>> *** > >>>>> > >>>>> The incubation status of Sedona can be found in [1]. The Apache > Sedona > >>>>> community has a discussion thread of graduating Sedona to a TLP [2], > >>>> passed > >>>>> the community vote [3]. The vote result was announced [4]. The PPMC > also > >>>>> passed a vote [5] to nominate Jia as the chair of Sedona TLP, and the > >>>>> result was also announced [6]. We had a discussion about Sedona > >>>> graduation > >>>>> in the incubator this September [12] and all issues have been fixed. > >>>>> > >>>>> The summary on the state of the community is listed below and the > self > >>>>> assessment (Apache Project Maturity Model) is in [7]. > >>>>> > >>>>> > >>>>> Sedona joined the Apache Incubator on July 19 2020. As of Dec 2 2022: > >>>>> > >>>>> ** Community **: > >>>>> > >>>>> * number of contributions (commits) since incubation: 394 > >>>>> - Overall: 1083 > >>>>> - 36% of the commits were added since incubation. > >>>>> > >>>>> * number of contributors (and growth since incubation): 51 > >>>>> - Overall: 86 > >>>>> - 59% of the contributors joined since incubation > >>>>> > >>>>> * email discussions since incubation: 1214 threads on > >>>> dev@sedona.apache.org > >>>>> > >>>>> * Number of monthly downloads: grow from 200K per month to 800K per > >>>> month. > >>>>> > >>>>> * Marked as critical project on PyPi as Sedona ranks among top 1% > most > >>>>> downloaded Python projects on PyPi. > >>>>> > >>>>> ** Project ** > >>>>> > >>>>> * releases: 7 > >>>>> > >>>>> * number of release managers: 3 (Jia Yu, Paweł Kociński, Kanchan > >>>> Chowdhury) > >>>>> > >>>>> * Apache Sedona website (sedona.apache.org) is compliant with ASF > >>>>> requirement [8]. The GitHub.com used in external resources are > compliant > >>>>> with ASF privacy requirement [9] > >>>>> > >>>>> ** Brands ,License, and Copyright ** > >>>>> > >>>>> * We submitted an application for the brand [10] and it has been > >>>>> reviewed an
Re: [DISCUSS] First Apache Sedona Community Call
I am sorry, but I am unable to attend. Will it be recorded? On Fri, Oct 7, 2022, 05:28 Jia Yu wrote: > Dear all, > > So far, we have 25 responses from the survey. The majority chose (1) Oct > 22 7 AM Pacific Time = 3 PM London = 11 PM Tokyo and (2) Zoom software for > the meeting. So let's have the community call on Oct 22! > > I will send a calendar invite with a Zoom link to everyone who has > registered. Anyone who wants to participate can still fill in the form > to register. > > Again, the link of the survey form is > https://www.surveymonkey.com/r/CKP9RR7 > > Sorry, I cannot post the Zoom link here since this is a public space and > might lead to Zoom bombing. > > Thanks, > Jia Yu > > > On Tue, Sep 27, 2022 at 9:52 PM Jia Yu wrote: > > > Hi Kengo, > > > > No problem at all! Looking forward to meeting you soon! > > > > Jia > > > > On Tue, Sep 27, 2022 at 3:28 AM Kengo Seki wrote: > > > >> Hi Jia, > >> > >> Thank you for planning this community call! > >> I'm not confident in real-time communication in English, but I'm > >> really looking forward to it. > >> > >> Kengo Seki > >> > >> 2022年9月25日(日) 4:07 Jia Yu : > >> > > >> > Dear all, > >> > > >> > I have created a survey form to decide the best meeting times from > you. > >> > Please fill in this survey and I will send you the meeting URL via > >> email: > >> > https://www.surveymonkey.com/r/CKP9RR7 > >> > > >> > Thank you for your participation! > >> > > >> > Jia > >> > > >> > On Thu, Sep 22, 2022 at 6:22 AM Pedro Mano Fernandes < > >> pedromor...@gmail.com> > >> > wrote: > >> > > >> > > Sounds good to me! > >> > > > >> > > On Mon, 19 Sept 2022 at 21:26, Jia Yu wrote: > >> > > > >> > > > Hi all, > >> > > > > >> > > > I am thinking of having our first ever Apache Sedona Community > >> Call. I > >> > > > believe it is a good time now to gather together all users and > >> > > contributors > >> > > > of Sedona in the same place and see each other. According to the > >> > > suggestion > >> > > > from the incubator community, we will use Sedona 1.3.0-incubating > >> as the > >> > > > final release before graduation. > >> > > > > >> > > > **Location** > >> > > > Online (Zoom, Microsoft Teams, or Google Meet) > >> > > > > >> > > > **Time** > >> > > > 1-hour, 2022, Oct 1st/8th/15th/22nd/29th (Saturday), 7 am Pacific > >> Time = > >> > > 3 > >> > > > pm London = 11 pm Tokyo > >> > > > > >> > > > I am using worldchatclock.com to find a time that overlaps most > >> > > continents > >> > > > in the world. Please feel free to give suggestions. > >> > > > > >> > > > **Agenda** > >> > > > 1. Round table introduction > >> > > > 2. The history of Sedona and recent major updates (Jia Yu) > >> > > > 3. Round table discussion about the roadmap of Sedona > >> > > > > >> > > > **Future plan** > >> > > > Based on the outcomes and turnout of this call, we probably will > >> make the > >> > > > community call as a recurring event (monthly or quarterly) or even > >> have > >> > > an > >> > > > in-person Sedona summit in the future (with the help of industrial > >> > > > sponsors) > >> > > > > >> > > > This is just a rough idea. Everyone, please feel free to give > >> feedback. > >> > > > Then, I will send a survey link to this email and determine the > >> exact > >> > > date > >> > > > and online meeting software. > >> > > > > >> > > > Thanks, > >> > > > Jia > >> > > > > >> > > > >> > > > >> > > -- > >> > > Pedro Mano Fernandes > >> > > > >> > > >
[jira] [Created] (SEDONA-134) Some Binder contrib notebooks are not working
Netanel Malka created SEDONA-134: Summary: Some Binder contrib notebooks are not working Key: SEDONA-134 URL: https://issues.apache.org/jira/browse/SEDONA-134 Project: Apache Sedona Issue Type: Bug Reporter: Netanel Malka I tried VectorAnalisisApacheSedona, and ApacheSedonaImageFilter. They throw exceptions. (e.g.: links to HDFS that do not exist) I think that the examples should work if someone wants to use them :) Also, the code contains code comments and does not contains comments about the commands. I think it should be in the standard of the other binder notebooks ([ApacheSedonaSQL.ipynb|https://hub-binder.mybinder.ovh/user/apache-incubator-sedona-99kmfiyu/notebooks/binder/ApacheSedonaSQL.ipynb] etc.) -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Graduate Apache Sedona (incubating) to a Top Level Project
+1 On Sun, 19 Jun 2022, 4:32 Mo Sarwat wrote: > +1 > > > On Sat, Jun 18, 2022 at 6:05 PM Jia Yu wrote: > > > Dear all in the community, > > > > I am starting a vote to graduate Apache Sedona (incubating) as a Top > Level > > Project. > > > > After a community discussion [1], we gathered a lot of feedback that > > the community supported Sedona to graduate from the Apache incubator > > as a top-level Apache project. > > > > Sedona has been incubating since 07-19-2020. For almost 2 years, the > > Sedona community has made great progress and grown rapidly under the > > guidance of our mentors. We believe the Sedona project has met the > > conditions for graduation. Project maturity model is detailed in [2], > > and the summary on the state of the community are listed below. > > > > Sedona joined the Apache Incubator on July 19 2020. As of May 29 2022 > > > > ***Community statistics*** > > > > - number of contributions (commits) since incubation > > 265 > > Overall: 965 > > 27% of the commits were added since incubation. > > > > - number of contributors (and growth since incubation) > > 40 > > Overall: 74 > > 54% of the contributors joined since incubation > > > > - email discussions since incubation: 798 threads on > dev@sedona.apache.org > > > > - releases: 5 > > > > - number of release managers: 2 (Jia Yu and Paweł Kociński) > > > > - affiliations of committers / PPMC members > > Paweł Kociński: GetInData (Poland) > > Netanel Malka: Sela Group (Israel) > > Jinxuan Wu: Bloomberg (US) > > Zongsi Zhang: Grab (Singapore) > > Jia Yu: Washington State University (US) > > Mohame Sarwat: Arizona State University (US) > > Yitao Li: SafeGraph (US) > > > > - new committers or PPMC members since incubation > > Yitao Li: SafeGraph > > > > - monthly downloads: > > over 500k from Maven Central, PyPi and CRAN. > > > > ***Code, License, and Copyright*** > > > > Sedona community maintains project code on GitHub, and all modules > > code is under Apache 2.0 license. We have reviewed all the > > dependencies and ensured they do not bring any license issues [3]. All > > the status files, license headers, and copyright are up to date. > > > > ***Quality*** > > > > Sedona community created 5 GitHub Workflows [4], which cover > > compilation, unit testing, integration testing, etc. All code patches > need > > to pass these > > pipelines first. In addition, Sedona has a matching Pull Request > > template and commit requirements to ensure that all commits have > > corresponding classification and traceability. > > > > ***Independence*** > > > > All of the 7 PPMC members are spread across 7 different > > organizations. > > > > For these reasons, we initiate this vote on whether to graduate Sedona > > from Apache Incubator, everyone from the Sedona community has the > > right to vote, please vote on graduating Sedona from Apache Incubator: > > > > [ ] +1 yes, I support Sedona to graduate from the Apache incubator and > > become a top-level Apache project. > > > > [ ] +0 no opinion > > > > [ ] -1 no, the Sedona project is not ready to graduate, because ... > > > > This vote will be open for at least 72 hours or until receive 3 +1 vote > > from PPMC. > > > > [1] https://lists.apache.org/thread/svd06kp5dojvw7pyk731tbrzkqfhjzbs > > [2] https://issues.apache.org/jira/browse/SEDONA-81 > > [3] https://github.com/apache/incubator-sedona/blob/master/LICENSE > > [4] > > https://github.com/apache/incubator-sedona/tree/master/.github/workflows > > >
[jira] [Created] (SEDONA-121) Add equivalent constructors left over from Spark to Flink
Netanel Malka created SEDONA-121: Summary: Add equivalent constructors left over from Spark to Flink Key: SEDONA-121 URL: https://issues.apache.org/jira/browse/SEDONA-121 Project: Apache Sedona Issue Type: Improvement Reporter: Netanel Malka Assignee: Netanel Malka The following constructors are missing from the Flink API: ST_LineStringFromText ST_LineFromText ST_Point ST_GeomFromText I will implement them using the Flink API. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (SEDONA-92) Build failed using Intellij IDEA
[
https://issues.apache.org/jira/browse/SEDONA-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17523892#comment-17523892
]
Netanel Malka commented on SEDONA-92:
-
I tried it, but I still have those issues.
The workaround for me is to exclude:
incubator-sedona/sql/src/test/scala/org/apache/sedona/sql/functions/collect
incubator-sedona/sql/src/test/scala/org/apache/sedona/sql/functions/geohash
But now I am facing new issues from Intellij (and a lot more errors like this
from Flink test packages):
{code:java}
java: package org.apache.sedona.flink.expressions does not exist {code}
And running from Maven:
{code:java}
Could not find artifact
org.apache.sedona:sedona-core-3.0_2.12:pom:1.2.1-incubating-SNAPSHOT in
snapshots-repo {code}
I would appreciate any thoughts you have on that.
> Build failed using Intellij IDEA
> -
>
> Key: SEDONA-92
> URL: https://issues.apache.org/jira/browse/SEDONA-92
> Project: Apache Sedona
> Issue Type: Task
> Reporter: Netanel Malka
>Priority: Normal
>
> Build using Intellij failed with the error message:
> {code:java}
> /home/netanel/IdeaProjects/incubator-sedona/sql/src/test/scala/org/apache/sedona/sql/functions/collect/TestStCollect.scala:22:8
> object GeometrySample is not a member of package org.apache.sedona.sql
> import org.apache.sedona.sql.{GeometrySample, TestBaseScala}{code}
> This makes it harder to run tests using the Intellij and enforce me to use
> maven which is not suitable for debugging and running tests.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: [VOTE] Release Apache Sedona 1.2.0-incubating-rc1
+1 Thanks, Netanel Malka On Wed, 23 Mar 2022 at 20:40, Mo Sarwat wrote: > +1 > > On Wed, Mar 23, 2022 at 11:38 AM Jia Yu wrote: > > > Dear all, > > > > +1 (binding) > > > > Thanks, > > Jia > > > > On Tue, Mar 22, 2022 at 11:44 PM Paweł Kociński > > wrote: > > > > > Hi all, > > > > > > This is a call for vote on Apache Sedona 1.2.0-incubating-rc1. Please > > refer > > > to the changes listed at the bottom of this email. > > > > > > Release notes: > > > > > > > > > https://github.com/apache/incubator-sedona/blob/sedona-1.2.0-incubating-rc1/docs/setup/release-notes.md > > > > > > Build instructions: > > > > > > > > > https://github.com/apache/incubator-sedona/blob/sedona-1.2.0-incubating-rc1/docs/setup/compile.md > > > > > > GitHub tag: > > > > > > > > > https://github.com/apache/incubator-sedona/releases/tag/sedona-1.2.0-incubating-rc1 > > > > > > GPG public key to verify the Release: > > > https://downloads.apache.org/incubator/sedona/KEYS > > > > > > Source code and binaries: > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/sedona/1.2.0-incubating-rc1/ > > > > > > The vote will be open for at least 72 hours or until a majority of at > > least > > > 3 "approve" PMC votes are cast > > > > > > Please vote accordingly: > > > > > > [ ] +1 approve > > > > > > [ ] +0 no opinion > > > > > > [ ] -1 disapprove with the reason > > > > > > Checklist for reference (because of DISCLAIMER-WIP, other checklist > items > > > are not blockers): > > > > > > [ ] Download links are valid. > > > > > > [ ] Checksums and PGP signatures are valid. > > > > > > [ ] DISCLAIMER is included. > > > > > > [ ] Source code artifacts have correct names matching the current > > release. > > > > > > For a detailed checklist please refer to: > > > > > > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > > > > > > > > > > > Changes according to the comments on the previous release > > > Original comment (Permalink from https://lists.apache.org/list.html): > > > https://lists.apache.org/thread/d2wbfqmmrnvd8wlfgyynt87xq8cs1k4v > > > > > > 1. NOTICE file > > > Fixed (see https://github.com/apache/incubator-sedona/pull/594) > > > > > > 2. headers for R workflow scripts > > > Added (see https://github.com/apache/incubator-sedona/pull/594) > > > > > > 3. checksum and signature name on Sedona website download page > > > Fixed (see https://sedona.apache.org/download/) > > > > > > Regards, > > > Paweł > > > > > > -- Best regards, Netanel Malka.
[jira] [Commented] (SEDONA-85) Add ST_GeoHash to Sedona flink streaming.
[ https://issues.apache.org/jira/browse/SEDONA-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509468#comment-17509468 ] Netanel Malka commented on SEDONA-85: - I can help with that. Do you think we shall create another module for that? or we can use the core module using scala? > Add ST_GeoHash to Sedona flink streaming. > - > > Key: SEDONA-85 > URL: https://issues.apache.org/jira/browse/SEDONA-85 > Project: Apache Sedona > Issue Type: Improvement >Reporter: Paweł Kociński >Assignee: Paweł Kociński >Priority: High > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (SEDONA-93) Add ST_GeomFromGeoJSON to Sedona flink
Netanel Malka created SEDONA-93: --- Summary: Add ST_GeomFromGeoJSON to Sedona flink Key: SEDONA-93 URL: https://issues.apache.org/jira/browse/SEDONA-93 Project: Apache Sedona Issue Type: Task Reporter: Netanel Malka Assignee: Netanel Malka -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (SEDONA-92) Build failed using Intellij IDEA
Netanel Malka created SEDONA-92:
---
Summary: Build failed using Intellij IDEA
Key: SEDONA-92
URL: https://issues.apache.org/jira/browse/SEDONA-92
Project: Apache Sedona
Issue Type: Task
Reporter: Netanel Malka
Build using Intellij failed with the error message:
{code:java}
/home/netanel/IdeaProjects/incubator-sedona/sql/src/test/scala/org/apache/sedona/sql/functions/collect/TestStCollect.scala:22:8
object GeometrySample is not a member of package org.apache.sedona.sql
import org.apache.sedona.sql.{GeometrySample, TestBaseScala}{code}
This makes it harder to run tests using the Intellij and enforce me to use
maven which is not suitable for debugging and running tests.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (SEDONA-85) Add ST_GeoHash to Sedona flink streaming.
[ https://issues.apache.org/jira/browse/SEDONA-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504399#comment-17504399 ] Netanel Malka commented on SEDONA-85: - [~jiayu] I am wondering if we can generalize the SQL functions and create something like a core functions that both spark and flink can use (and another modules will be use in the future). I thought about putting the actual implementation of the geo functions in a core module, and Spark and Flink will only call those functions. What do you think? [~imbruced] Is that means that we will need to rewrite the python API? > Add ST_GeoHash to Sedona flink streaming. > - > > Key: SEDONA-85 > URL: https://issues.apache.org/jira/browse/SEDONA-85 > Project: Apache Sedona > Issue Type: Improvement >Reporter: Paweł Kociński >Assignee: Paweł Kociński >Priority: High > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (SEDONA-85) Add ST_GeoHash to Sedona flink streaming.
[ https://issues.apache.org/jira/browse/SEDONA-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503365#comment-17503365 ] Netanel Malka commented on SEDONA-85: - [~imbruced] [~jiayu] Would we like to add all the functions that Spark supports to Flink? If the answer is yes, I would like to help with that. > Add ST_GeoHash to Sedona flink streaming. > - > > Key: SEDONA-85 > URL: https://issues.apache.org/jira/browse/SEDONA-85 > Project: Apache Sedona > Issue Type: Improvement >Reporter: Paweł Kociński >Assignee: Paweł Kociński >Priority: High > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (SEDONA-52) Create Osm pbf loader for Apache Sedona.
[ https://issues.apache.org/jira/browse/SEDONA-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17464953#comment-17464953 ] Netanel Malka commented on SEDONA-52: - [~jiayu] , [~imbruced] Both of the projects are using the MIT license, which AFAIK is very premissive. Also, I saw in the official [ASF 3RD PARTY LICENSE POLICY|https://www.apache.org/legal/resolved.html#highlevel] that we can use them freely. Am I understanding that correctly? > Create Osm pbf loader for Apache Sedona. > > > Key: SEDONA-52 > URL: https://issues.apache.org/jira/browse/SEDONA-52 > Project: Apache Sedona > Issue Type: New Feature >Reporter: Paweł Kociński >Assignee: Paweł Kociński >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (SEDONA-52) Create Osm pbf loader for Apache Sedona.
[ https://issues.apache.org/jira/browse/SEDONA-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456964#comment-17456964 ] Netanel Malka commented on SEDONA-52: - [~imbruced] Are you working on that? I found those packages which can help us to implement that: Python: [https://github.com/HTenkanen/pyrosm/blob/master/pyrosm/pyrosm.py] Sacla: [https://github.com/simplexspatial/osm4scala] > Create Osm pbf loader for Apache Sedona. > > > Key: SEDONA-52 > URL: https://issues.apache.org/jira/browse/SEDONA-52 > Project: Apache Sedona > Issue Type: New Feature >Reporter: Paweł Kociński >Assignee: Paweł Kociński >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (SEDONA-28) Add WKB serializer in RDD and SQL API and let the user choose the SerDe
[
https://issues.apache.org/jira/browse/SEDONA-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456960#comment-17456960
]
Netanel Malka commented on SEDONA-28:
-
[~jiayu] I saw that:
{code:java}
Reverts #516
PR 516 for SEDONA-28 requires a more thorough test under the cluster mode. Will
not include it in Sedona 1.1.0.
{code}
This issue is still relevant?
Can we check the bug on a local cluster?
There are any actions we can take to prevent from that happening? (tests on a
local cluster or something like that)
> Add WKB serializer in RDD and SQL API and let the user choose the SerDe
>
>
> Key: SEDONA-28
> URL: https://issues.apache.org/jira/browse/SEDONA-28
> Project: Apache Sedona
> Issue Type: Task
> Reporter: Netanel Malka
> Assignee: Netanel Malka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Add WKB serializer as an optional serializer and use the old Serde as the
> default SerDe for both Core and SQL. The user should only use it if they use
> geometries that are currently not supported by the old Serde.
> The user should be able to choose the serializer as follows
> .config("spark.serializer", classOf[KryoSerializer].getName) //
> org.apache.spark.serializer.KryoSerializer
> .config("spark.kryo.registrator", classOf[SedonaKryoRegistrator].getName)
> .config("spark.serializer", classOf[KryoSerializer].getName) //
> org.apache.spark.serializer.KryoSerializer
> .config("spark.kryo.registrator", classOf[SedonaWKBKryoRegistrator].getName)
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: [VOTE] Release Apache Sedona 1.1.1-incubating-rc1
+1 Great work! On Sun, Nov 14, 2021, 20:13 Adam Binford wrote: > +1 (non-binding) > > On Sun, Nov 14, 2021, 3:00 AM Jia Yu wrote: > >> Hi all, This is a call for vote on Apache Sedona 1.1.1-incubating-rc1. >> Please refer to the changes listed at the bottom of this email. Release >> notes: >> >> https://github.com/apache/incubator-sedona/blob/sedona-1.1.1-incubating-rc1/docs/download/release-notes.md >> Build instructions: >> >> https://github.com/apache/incubator-sedona/blob/sedona-1.1.1-incubating-rc1/docs/download/compile.md >> GitHub tag: >> >> https://github.com/apache/incubator-sedona/releases/tag/sedona-1.1.1-incubating-rc1 >> GPG public key to verify the Release: >> https://dist.apache.org/repos/dist/dev/incubator/sedona/KEYS Source code >> and binaries: >> >> https://dist.apache.org/repos/dist/dev/incubator/sedona/1.1.1-incubating-rc1/ >> The vote will be open for at least 72 hours or until a majority of at >> least >> 3 "approve" PMC votes are cast Please vote accordingly: [ ] +1 approve [ ] >> +0 no opinion [ ] -1 disapprove with the reason Checklist for reference >> (because of DISCLAIMER-WIP, other checklist items are not blockers): [ ] >> Download links are valid. [ ] Checksums and PGP signatures are valid. [ ] >> DISCLAIMER is included. [ ] Source code artifacts have correct names >> matching the current release. For a detailed checklist please refer to: >> >> https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist >> >
Re: [VOTE] Release Apache Sedona 1.1.0-incubating-rc1
+1 @Paweł Kociński Yes. but Jia revert the PR so it won't be included in that release. I will try to look into this soon. On Fri, 1 Oct 2021 at 08:18, Paweł Kociński wrote: > Hi, > Is this issue still remain the problem ? > https://github.com/apache/incubator-sedona/commit/a0ed49da0b62ce591d858e5dccd5cebb69a8e695#comments > > Regards, > Paweł > > czw., 30 wrz 2021 o 15:44 Mo Sarwat napisał(a): > >> +1 >> >> On Wed, Sep 29, 2021 at 9:38 PM Jia Yu wrote: >> >>> CC several key members in the community >>> >>> >>> >>> On Wed, Sep 29, 2021 at 9:31 PM Jia Yu wrote: >>> >>> > Hi all, >>> > >>> > This is a call for vote on Apache Sedona 1.1.0-incubating-rc1. Please >>> > refer to the changes listed at the bottom of this email. >>> > >>> > Release notes: >>> > >>> > >>> https://github.com/apache/incubator-sedona/blob/sedona-1.1.0-incubating-rc1/docs/download/release-notes.md >>> > >>> > Build instructions: >>> > >>> > >>> https://github.com/apache/incubator-sedona/blob/sedona-1.1.0-incubating-rc1/docs/download/compile.md >>> > >>> > GitHub tag: >>> > >>> > >>> https://github.com/apache/incubator-sedona/releases/tag/sedona-1.1.0-incubating-rc1 >>> > >>> > GPG public key to verify the Release: >>> > https://dist.apache.org/repos/dist/dev/incubator/sedona/KEYS >>> > >>> > Source code and binaries: >>> > >>> > >>> https://dist.apache.org/repos/dist/dev/incubator/sedona/1.1.0-incubating-rc1/ >>> > >>> > The vote will be open for at least 72 hours or until a majority of at >>> > least 3 "approve" PMC votes are cast >>> > >>> > Please vote accordingly: >>> > >>> > [ ] +1 approve >>> > >>> > [ ] +0 no opinion >>> > >>> > [ ] -1 disapprove with the reason >>> > >>> > Checklist for reference (because of DISCLAIMER-WIP, other checklist >>> items >>> > are not blockers): >>> > >>> > [ ] Download links are valid. >>> > >>> > [ ] Checksums and PGP signatures are valid. >>> > >>> > [ ] DISCLAIMER is included. >>> > >>> > [ ] Source code artifacts have correct names matching the current >>> release. >>> > >>> > For a detailed checklist please refer to: >>> > >>> > >>> https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist >>> > >>> > >>> > Changes according to the comments on the previous release >>> > Original comment (Permalink from https://lists.apache.org/list.html): >>> > >>> > >>> https://lists.apache.org/thread.html/rcc3252c2afda7761afb205eb00a2f8a5d71256fc7f4d631efc879629%40%3Cgeneral.incubator.apache.org%3E >>> > >>> > 1. Fix the header issue in the BSD license >>> > This has been fixed. >>> > >>> > [1] >>> > >>> https://github.com/apache/incubator-sedona/blob/sedona-1.1.0-incubating-rc1/zeppelin/index.js >>> > [2] >>> > >>> https://github.com/apache/incubator-sedona/blob/sedona-1.1.0-incubating-rc1/licenses/LICENSE-zeppelin-helium-plugin >>> > >>> > 2. Add the license for scalastyle_config.xml >>> > >>> > [3] >>> > >>> https://github.com/apache/incubator-sedona/blob/sedona-1.1.0-incubating-rc1/LICENSE#L210 >>> > >>> > 3. Replace NASA HDF data with some alternatives to avoid the legal >>> issue >>> > >>> > All NASA data has been removed from the repo. >>> > >>> >> -- Best regards, Netanel Malka.
Re: [VOTE] Release Apache Sedona 1.0.1-incubating-rc1
+1 On Sat, 22 May 2021 at 07:34, Mo Sarwat wrote: > +1 > > On Tue, May 18, 2021 at 9:57 PM Jia Yu wrote: > > > Hi all, > > > > This is a call for vote on Apache Sedona 1.0.1-incubating-rc1. Please > refer > > to the changes listed at the bottom of this email. > > > > Release notes: > > > > > https://github.com/apache/incubator-sedona/blob/sedona-1.0.1-incubating-rc1/docs/download/release-notes.md > > > > Build instructions: > > > > > https://github.com/apache/incubator-sedona/blob/sedona-1.0.1-incubating-rc1/docs/download/compile.md > > > > GitHub tag: > > > > > https://github.com/apache/incubator-sedona/releases/tag/sedona-1.0.1-incubating-rc1 > > > > GPG public key to verify the Release: > > https://dist.apache.org/repos/dist/dev/incubator/sedona/KEYS > > > > Source code and binaries: > > > > > https://dist.apache.org/repos/dist/dev/incubator/sedona/1.0.1-incubating-rc1/ > > > > The vote will be open for at least 72 hours or until a majority of at > least > > 3 +1 PMC votes are cast > > > > Please vote accordingly: > > > > [ ] +1 approve > > > > [ ] +0 no opinion > > > > [ ] -1 disapprove with the reason > > > > Checklist for reference (because of DISCLAIMER-WIP, other checklist items > > are not blockers): > > > > [ ] Download links are valid. > > > > [ ] Checksums and PGP signatures are valid. > > > > [ ] DISCLAIMER is included. > > > > [ ] Source code artifacts have correct names matching the current > release. > > > > For a detailed checklist please refer to: > > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > > > > > Changes according to the comments of Justin Mclean on the > 1.0.0-incubating > > release > > Original comment URL: > > > > > https://lists.apache.org/thread.html/r828873cbb2685dcfb0719680f3aac6dbf982720fcd9cd5f69a26ec55%40%3Cgeneral.incubator.apache.org%3E > > > > 1. There are some test files I think I like to know where they come from > > e.g. county_small_wkb.tsv and what license the contents are under. > > > > License for test data has been added to Sedona license > > > > > https://github.com/apache/incubator-sedona/blob/sedona-1.0.1-incubating-rc1/LICENSE > > > > 2. The LICENSE here seems odd why does it have "Copyright (c) 2019-2020, > > Apache Sedona” in it? > > > > The issue has been fixed. The license for zeppelin plugin has been added > to > > LICENSE as well. > > > > > https://github.com/apache/incubator-sedona/blob/sedona-1.0.1-incubating-rc1/LICENSE > > > > 3. Please put instructions on how to build in the release, instructions > can > > change over time so pointing to a URL may not be helpful when trying to > > build older versions. > > > > Build instruction has been added to svn/dist/dev > > > > > https://dist.apache.org/repos/dist/dev/incubator/sedona/1.0.1-incubating-rc1/ > > > -- Best regards, Netanel Malka.
[jira] [Closed] (SEDONA-17) Replace geometry serializer in RDD API with the WKB serializer
[ https://issues.apache.org/jira/browse/SEDONA-17?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Netanel Malka closed SEDONA-17. --- Resolution: Won't Do Will be part of SEDONA-28 > Replace geometry serializer in RDD API with the WKB serializer > -- > > Key: SEDONA-17 > URL: https://issues.apache.org/jira/browse/SEDONA-17 > Project: Apache Sedona > Issue Type: Task > Reporter: Netanel Malka > Assignee: Netanel Malka >Priority: Normal > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > In the Sedona SQL module, we are using the WKB serializer instead of the > Shape serializer because of an old bug. > Now, we want to replace also the serializer in the RDD API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SEDONA-17) Replace geometry serializer in RDD API with the WKB serializer
[ https://issues.apache.org/jira/browse/SEDONA-17?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17310987#comment-17310987 ] Netanel Malka commented on SEDONA-17: - Close this issue and open SEDONA-28 as we decided to implement both of the serdes and let the user decide what to use. > Replace geometry serializer in RDD API with the WKB serializer > -- > > Key: SEDONA-17 > URL: https://issues.apache.org/jira/browse/SEDONA-17 > Project: Apache Sedona > Issue Type: Task > Reporter: Netanel Malka > Assignee: Netanel Malka >Priority: Normal > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > In the Sedona SQL module, we are using the WKB serializer instead of the > Shape serializer because of an old bug. > Now, we want to replace also the serializer in the RDD API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (SEDONA-28) Add WKB serializer in RDD and SQL API and let the user choose the SerDe
Netanel Malka created SEDONA-28:
---
Summary: Add WKB serializer in RDD and SQL API and let the user
choose the SerDe
Key: SEDONA-28
URL: https://issues.apache.org/jira/browse/SEDONA-28
Project: Apache Sedona
Issue Type: Task
Reporter: Netanel Malka
Assignee: Netanel Malka
Add WKB serializer as an optional serializer and use the old Serde as the
default SerDe for both Core and SQL. The user should only use it if they use
geometries that are currently not supported by the old Serde.
The user should be able to choose the serializer as follows
.config("spark.serializer", classOf[KryoSerializer].getName) //
org.apache.spark.serializer.KryoSerializer
.config("spark.kryo.registrator", classOf[SedonaKryoRegistrator].getName)
.config("spark.serializer", classOf[KryoSerializer].getName) //
org.apache.spark.serializer.KryoSerializer
.config("spark.kryo.registrator", classOf[SedonaWKBKryoRegistrator].getName)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (SEDONA-17) Replace geometry serializer in RDD API with the WKB serializer
Netanel Malka created SEDONA-17: --- Summary: Replace geometry serializer in RDD API with the WKB serializer Key: SEDONA-17 URL: https://issues.apache.org/jira/browse/SEDONA-17 Project: Apache Sedona Issue Type: Task Reporter: Netanel Malka Assignee: Netanel Malka In the Sedona SQL module, we are using the WKB serializer instead of the Shape serializer because of an old bug. Now, we want to replace also the serializer in the RDD API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Develop new features/fix bugs
Thanks, Jia. I think I will start with the first issue. I will create a JIRA Ticket. By the way, I tried to work on Intellij IDEA and I can't make the project work. There are dependencies issues that only appear on Intellij and not when using maven. Hence, I am working with VS code which appears to work well. Do you work with Intellij as a code editor on the Sedona project? Thanks. On Thu, 18 Feb 2021 at 23:02, Paweł Kociński wrote: > Hi Jia, > Do we have some estimates about next release ? I think it is good idea to > create JIRA tasks from that link > > https://lists.apache.org/thread.html/r4ebf7c253fff77900cac8b6ad478c8f4808b707062dd8c6b40853a6f%40%3Cdev.sedona.apache.org%3E > . > I can focus on point Implement loading geospatial data sources (geojson, > osm) and postgis direct support. > Regards, > Paweł > > czw., 18 lut 2021 o 06:38 Jia Yu napisał(a): > > > H Netanel, > > > > Thanks for asking. I think there are a couple of issues to be improved, > as > > listed here: > > > https://lists.apache.org/thread.html/r4ebf7c253fff77900cac8b6ad478c8f4808b707062dd8c6b40853a6f%40%3Cdev.sedona.apache.org%3E > > > > On the list given by Pawel and my thoughts, I think the most important > > thing to be fixed is that (sorted by importance): > > > > 1. Replace the object serializer in RDD API with the WKB serializer you > > wrote before, currently only DataFrame API uses WKB based serializer. In > > Sedona-core, there is a small benchmark that can test the performance of > > the WKB serializer. > > 2. Allow broadcast join for big+small data frames, or at least give users > > a way to force the broadcast join. Currently, once the optimized Sedona > > join is enabled, the broadcast join is automatically disabled. > > 3. Allow null spatial objects in SpatialRDD and DataFrame. Or at least > > give users a meaningful exception: > > https://issues.apache.org/jira/browse/SEDONA-4 > > > > Pawel was working on some improvements for GeoJSON loading. So we can > > leave it for him. > > > > Do you have any other issues you want to work on? > > > > Thanks, > > Jia > > > > On Wed, Feb 17, 2021 at 5:35 AM Netanel Malka wrote: > > > >> Hi Jia, > >> Are there any features that we want to develop? or fix some bugs? > >> > >> My free time now allows me to make some contributions to the project. > >> > >> Thanks, > >> > >> > -- Best regards, Netanel Malka.
Develop new features/fix bugs
Hi Jia, Are there any features that we want to develop? or fix some bugs? My free time now allows me to make some contributions to the project. Thanks,
Re: [Bug][Python] Missing Java Class?
Hi Gregory,
Can you please try to install the jars on the Databricks Cluster?
For example:
On clusters -> choose your cluster -> libraries -> install new:
1.Coordinates: org.geotools:gt-main:24.0
2.repo: https://repo.osgeo.org/repository/release/
I successfully did it.
Please let me know if it solves your problem
On 2021/02/10 13:16:50, Grégory Dugernier wrote:
> Thank you for the quick reply!
>
> It seems my particular situation is a bit more complex than that, since I'm
> running the notebook on a Databricks cluster, and the default spark config
> doesn't seem to allow for more jar repositories (GeoTools isn't on Maven
> Central), nor does creating a new SparkSession appears to work. I've tried
> to download the jars and add them manually to the cluster but it doesn't
> seem to work either. But at least I know where the issue's at!
>
> Thanks again for your help,
> Regards
>
> On Wed, 10 Feb 2021 at 12:22, Jia Yu wrote:
>
> > Hi Gregory,
> >
> > Thanks for letting us know. This is not a bug. We cannot include GeoTools
> > jars due to license issues. But indeed we forgot to update the docs and
> > jupyter notebook examples. I just updated them. Please read them here:
> >
> >
> > https://github.com/apache/incubator-sedona/blob/master/python/ApacheSedonaSQL.ipynb
> >
> > (Make sure you disable the browser cache or open it in an incognito
> > window) http://sedona.apache.org/download/overview/#install-sedona-python
> >
> > In short, you need to add the following coordinates in the notebook:
> >
> > spark = SparkSession. \ builder. \ appName('appName'). \ config(
> > "spark.serializer", KryoSerializer.getName). \ config(
> > "spark.kryo.registrator", SedonaKryoRegistrator.getName). \ config(
> > "spark.jars.repositories", 'https://repo.osgeo.org/repository/release,' '
> > https://download.java.net/maven/2'). \ config('spark.jars.packages',
> > 'org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.0-incubating,'
> > 'org.geotools:gt-main:24.0,' 'org.geotools:gt-referencing:24.0,'
> > 'org.geotools:gt-epsg-hsql:24.0'). \ getOrCreate()
> >
> > On Wed, Feb 10, 2021 at 2:35 AM Grégory Dugernier wrote:
> >
> >> Hello,
> >>
> >> I've been trying to run Sedona for Python on Databricks for 2 days and I
> >> think I've stumbled upon a bug.
> >>
> >> *Configuration*:
> >>
> >>- Spark 3.0.1
> >>- Scala 2.12
> >>- Python 3.7
> >>
> >> *Librairies*:
> >>
> >>- apache-sedona (from PyPi)
> >>- org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.0-incubating
> >>(from Maven)
> >>
> >> *What I'm trying to do:*
> >>
> >> I'm trying to load a series of Shapefiles files into a dataframe for
> >> geospatial analysis. See code snippet below, based of your example
> >> notebook
> >> <
> >> https://github.com/apache/incubator-sedona/blob/master/python/ApacheSedonaCore.ipynb
> >> >
> >>
> >>
> >> > from sedona.core.formatMapper.shapefileParser import ShapefileReader
> >> > from sedona.register import SedonaRegistrator
> >> > from sedona.utils.adapter import Adapter
> >> >
> >> > SedonaRegistrator.registerAll(spark)
> >> > shape_rdd = ShapefileReader.readToGeometryRDD(spark.sparkContext,
> >> > file_name)
> >> > df = Adapter.toDf(shape_rdd, spark)
> >> >
> >>
> >> *Bug*:
> >>
> >> The ShapefileReader.readToGeometryRDD() currently throws the following
> >> error:
> >>
> >> > Py4JJavaError: An error occurred while calling
> >> >
> >> z:org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD.
> >> > : java.lang.NoClassDefFoundError:
> >> org/opengis/referencing/FactoryException
> >> > at
> >> >
> >> org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD(ShapefileReader.java:79)
> >> > at
> >> >
> >> org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD(ShapefileReader.java:66)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> >> >
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> > at
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> > at java.lang.reflect.Method.invoke(Method.java:498) at
> >> > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at
> >> > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380) at
> >> > py4j.Gateway.invoke(Gateway.java:295) at
> >> > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at
> >> > py4j.commands.CallCommand.execute(CallCommand.java:79) at
> >> > py4j.GatewayConnection.run(GatewayConnection.java:251) at
> >> > java.lang.Thread.run(Thread.java:748) Caused by:
> >> > java.lang.ClassNotFoundException:
> >> org.opengis.referencing.FactoryException
> >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> >> > java.lang.ClassLoader.loadClass(ClassLoader.java:419) at
> >> >
> >> com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader.loadClass(ClassLoaders.scala:151)
> >> > at
Re: [DISCUSS] Put all GeoTools jars into a package on Maven Central
I think that we can make it work on Databricks without any changes. After creating a cluster on Databricks, the user can install the geotools packages and provide the osego *(or any other repo) explicitly.* As you can see in the picture: [image: image.png] I can provide the details on how to install it. I think it will solve the problem. What do you think? On Thu, 11 Feb 2021 at 12:24, Jia Yu wrote: > Hi folks, > > As you can see from the recent discussion in the mailing list > <[Bug][Python] Missing Java class>, in Sedona 1.0.0, because those LGPL > GeoTools jars are not on Maven Central (only in OSGEO repo), Databricks > cannot get GeoTools jars. > > I believe this will cause lots of trouble to our future Python users. > Reading Shapefiles and do CRS transformation are big selling points for > Sedona. > > The easiest way to fix this, without violating ASF policy, is that I will > publish a GeoTools wrapper on Maven Central using the old GeoSpark group > ID: https://mvnrepository.com/artifact/org.datasyslab > > For example, org.datasyslab:geotools-24-wrapper:1.0.0 > > 1. This GeoTools wrapper does nothing but brings the GeoTools jars needed > by Sedona to Maven Central. > 2. When the Python user calls Sedona, they can add one more > package: org.datasyslab:geotools-24-wrapper:1.0.0 > > Another good thing is that: this does not require a new source code > release from Sedona. We only need to update the website and let the users > know how to call it. > > Any better ideas? > > Thanks, > Jia > > > -- Best regards, Netanel Malka.
Re: [VOTE] Apache Sedona (incubating) rc1
; >> > Please vote accordingly: > >> > > >> > [ ] +1 approve > >> > > >> > [ ] +0 no opinion > >> > > >> > [ ] -1 disapprove with the reason > >> > > >> > Checklist for reference (because of DISCLAIMER-WIP, other checklist > >> items > >> > are not blockers): > >> > > >> > [ ] Download links are valid. > >> > > >> > [ ] Checksums and PGP signatures are valid. > >> > > >> > [ ] DISCLAIMER is included. > >> > > >> > [ ] Source code artifacts have correct names matching the current > >> release. > >> > > >> > For a detailed checklist please refer to: > >> > > >> > > >> > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > >> > > >> > Changes from the last failed vote (let's call it rc0 :-) ): > >> > 1. Version changed from "1.0.0-incubator" to "1.0.0-incubating" > >> > 2. Source code and binary now hosted on dist/dev/incubator/sedona/ > >> > 3. Source code and binary now are signed by RSA-4096 key with > >> > ji...@apache.org > >> > 4. Source code and binary now use sha512 checksum > >> > 5. Binaries are not staged on repository.apache.org yet. I will stage > >> them > >> > after the vote passes. > >> > 6. The compressed source code size changed from 600+ MB to 100+ MB, by > >> > remove duplicate test data. Will find a way to further in the future > >> > releases. > >> > 7. GeoTools (LGPL license, ASF Category X license) binaries have been > >> > completed removed from Sedona binaries. Users have to compile the > source > >> > code by themselves if they want to use ShapefileReader and CRS > >> > Transformation. > >> > > >> > @fengji...@gmail.com Von, the GitHub tag is > >> called > >> > "sedona-1.0.0-incubating-rc1" which has "sedona" prefix. This is > because > >> > our GitHub tags include old tags for GeoSpark. To differentiate Sedona > >> tags > >> > from old tags, I intend to add the "sedona" prefix. > >> > > >> > Thank you very much! > >> > Jia > >> > > >> > > > -- Best regards, Netanel Malka.
Re: [VOTE] Release Apache Sedona (incubating) 1.0.0
Great Job!! +1 Netanel Malka On 2021/01/07 10:16:11, Jia Yu wrote: > Hi All, > > After a fruitful discussion about our first Apache Sedona release > 1.0.0-incubator, the release has been created. This is a call for vote to > release Apache Sedona (incubating) 1.0.0. > > Note that: the current sha1 and checksum verification of Sedona will > require us to manually download artifact jars, sha1, asc from > repository.apache.org 12 times each. It is very annoying. Please let me > know if you have any suggestions to speed up the process. > > Release note: > https://sedona.staged.apache.org/download/GeoSpark-All-Modules-Release-notes/#sedona-100 > > Build instructions: > https://sedona.staged.apache.org/download/compile/ > > Git tag: > https://github.com/apache/incubator-sedona/releases/tag/sedona-1.0.0-incubator > > Maven staging repository (search for "sedona", 12 artifacts in total): > https://repository.apache.org/#stagingRepositories > > Release Commit ID: > https://github.com/apache/incubator-sedona/commit/29f897bbcaf65aa5b5b28ea4e93c6b7e783d83a6 > > GPG public key to verify the Release: > https://gist.githubusercontent.com/jiayuasu/8b6cc263c538148490f7a090ee1dbeab/raw/09d3821e2794ddf22542f74c0db5195cf5c1ff49/gpg-public-key.gpg > > The vote will be open for at least 72 hours or until a majority of at least > 3 +1 PMC votes are cast > > Please vote accordingly: > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > Checklist for reference (because of DISCLAIMER-WIP, other checklist items > are not blockers): > > [ ] Download links are valid. > > [ ] Checksums and PGP signatures are valid. > > [ ] DISCLAIMER is included. > > [ ] Source code artifacts have correct names matching the current release. > > For a detailed checklist please refer to: > https://cwiki.apache.org/confluence/display/INCUBATOR/Incubator+Release+Checklist > > To verify the checksum, > 1. open https://repository.apache.org > 2 (1) click each release jar (12 in total) to see its current SHA1 (under > Artifact tab) (2) download .jar.sha1 to see the content of the uploaded > sha1. This two should match > > To verify the GPG key (12 in total), > gpg --import the-key-file > gpg --verify xxx.jar.asc xxx.jar > > You should see something like "gpg: Good signature from "Jia Yu (Arizona > State University Data Systems Lab) " gpg: WARNING: This > key is not certified with a trusted signature!" > > Thanks, > Jia >
Re: Sedona 1.0 Maven release is staged
Hi Jia, I succeeded to run the code both from the terminal and from IntelliJ. I needed a new project for the examples in IntelliJ, and not to add the module to the existing Sedona project. On Tue, 5 Jan 2021 at 08:53, Jia Yu wrote: > Hi Netanel, > > Thank you! Please try it again. If you can get it done, I will start a vote > thread in our community and then pass it to the incubator. > > Jia > > On Mon, Jan 4, 2021 at 10:38 AM Netanel Malka > wrote: > > > I tried it again with no success. > > Now, I want to try to remove the .iv2 folder and try it again, I found it > > in some posts on google. > > > > I will update soon. > > > > Thanks. > > > > On Sat, 2 Jan 2021 at 04:16, Jia Yu wrote: > > > > > Hi Netanel, > > > > > > The reason why I use sbt is that, when I use Maven to run SparkSQL 3.0 > + > > > Sedona in IntelliJ, it always prompts an Exception like "NoClassFound: > > > jackson-data-bind: noMappingException." This is because Spark 3.0 uses > > > inconsistent com.fasterxml.jackson version (2.10.0 vs some Spark > > > dependencies use 2.9.x). Maven always packages the 2.9.x for test no > > matter > > > how I change the pom.xml in example projects. > > > > > > If I use SBT, all exceptions are gone. The "cannot reparse for > SNAPSHOT" > > > issue in SBT is a warning, not an error. so it is not a blocker. But I > am > > > not sure where another error comes from. > > > > > > In fact, I have tested all example projects (with Sedona SNAPSHOTs) in > > > IntelliJ and CLI. I just added the GitHub action CI for the example > > > projects. It also passed: > > > https://github.com/apache/incubator-sedona/actions/runs/456989056 > > > > > > Do you mind pull the master branch again and run the three projects in > > CLI > > > and IntelliJ? See > > > > > > > > > https://github.com/apache/incubator-sedona/blob/master/.github/workflows/example.yml#L35 > > > > > > Thanks, > > > Jia > > > > > > > > > > > > > > > On Fri, Jan 1, 2021 at 1:14 AM Netanel Malka > > wrote: > > > > > > > Hi Jia, > > > > I tried to build the Demo project and I have a dependency issue. > > > > I tried using sbt assembly and from IntelliJ. > > > > > > > > For example, build error for Sedona SQL (None of them succeeded in > > > build): > > > > https://pastebin.pl/view/15882089 > > > > > > > > It looks like a bug in SBT regards to SNAPSHOT jars that don't > handle: > > > > https://github.com/sbt/sbt/issues/3404#issuecomment-400641680 > > > > and also here: > > > > https://github.com/sbt/sbt/issues/2687 > > > > > > > > There is any special reason why did you choose to use sbt instead of > > > maven? > > > > > > > > > > > > > > > > > > > > On Thu, 31 Dec 2020 at 23:27, Jia Yu wrote: > > > > > > > > > Dear Felix, > > > > > > > > > > Thanks for your great comments! > > > > > > > > > > I have fixed all three comments accordingly, Since we have > > > disclaim-WIP, > > > > I > > > > > think we don't have to re-stage the releases. > > > > > > > > > > Thanks, > > > > > Jia > > > > > > > > > > > > > > > > > > > > On Wed, Dec 30, 2020 at 10:54 PM Felix Cheung < > > felixche...@apache.org> > > > > > wrote: > > > > > > > > > > > Release review. > > > > > > > > > > > > Note: with DISCLAIMER-WIP, these are not blockers to release. > > > > > > > > > > > > website: > > > > > > 1. the website can add more links as described in this section: > > > > > > https://www.apache.org/foundation/marks/pmcs#navigation > > > > > > 2. might need to add trademark attributions > > > > > > https://www.apache.org/foundation/marks/pmcs#attributions > > > > > > > > > > > > project source: > > > > > > 1. For LICENSE, we should add RAT check > > > > > > https://creadur.apache.org/rat/#What_Is_Rat > > > > > > 2. This file seems to have an irregular header: > > > > > > /incubator-sedona/zeppelin/index.
Re: Sedona 1.0 Maven release is staged
I tried it again with no success. Now, I want to try to remove the .iv2 folder and try it again, I found it in some posts on google. I will update soon. Thanks. On Sat, 2 Jan 2021 at 04:16, Jia Yu wrote: > Hi Netanel, > > The reason why I use sbt is that, when I use Maven to run SparkSQL 3.0 + > Sedona in IntelliJ, it always prompts an Exception like "NoClassFound: > jackson-data-bind: noMappingException." This is because Spark 3.0 uses > inconsistent com.fasterxml.jackson version (2.10.0 vs some Spark > dependencies use 2.9.x). Maven always packages the 2.9.x for test no matter > how I change the pom.xml in example projects. > > If I use SBT, all exceptions are gone. The "cannot reparse for SNAPSHOT" > issue in SBT is a warning, not an error. so it is not a blocker. But I am > not sure where another error comes from. > > In fact, I have tested all example projects (with Sedona SNAPSHOTs) in > IntelliJ and CLI. I just added the GitHub action CI for the example > projects. It also passed: > https://github.com/apache/incubator-sedona/actions/runs/456989056 > > Do you mind pull the master branch again and run the three projects in CLI > and IntelliJ? See > > https://github.com/apache/incubator-sedona/blob/master/.github/workflows/example.yml#L35 > > Thanks, > Jia > > > > > On Fri, Jan 1, 2021 at 1:14 AM Netanel Malka wrote: > > > Hi Jia, > > I tried to build the Demo project and I have a dependency issue. > > I tried using sbt assembly and from IntelliJ. > > > > For example, build error for Sedona SQL (None of them succeeded in > build): > > https://pastebin.pl/view/15882089 > > > > It looks like a bug in SBT regards to SNAPSHOT jars that don't handle: > > https://github.com/sbt/sbt/issues/3404#issuecomment-400641680 > > and also here: > > https://github.com/sbt/sbt/issues/2687 > > > > There is any special reason why did you choose to use sbt instead of > maven? > > > > > > > > > > On Thu, 31 Dec 2020 at 23:27, Jia Yu wrote: > > > > > Dear Felix, > > > > > > Thanks for your great comments! > > > > > > I have fixed all three comments accordingly, Since we have > disclaim-WIP, > > I > > > think we don't have to re-stage the releases. > > > > > > Thanks, > > > Jia > > > > > > > > > > > > On Wed, Dec 30, 2020 at 10:54 PM Felix Cheung > > > wrote: > > > > > > > Release review. > > > > > > > > Note: with DISCLAIMER-WIP, these are not blockers to release. > > > > > > > > website: > > > > 1. the website can add more links as described in this section: > > > > https://www.apache.org/foundation/marks/pmcs#navigation > > > > 2. might need to add trademark attributions > > > > https://www.apache.org/foundation/marks/pmcs#attributions > > > > > > > > project source: > > > > 1. For LICENSE, we should add RAT check > > > > https://creadur.apache.org/rat/#What_Is_Rat > > > > 2. This file seems to have an irregular header: > > > > /incubator-sedona/zeppelin/index.js > > > > 3. These code file should have Apache v2 header: > > > > /python/build_wheel.sh > > > > /spark-version-converter.py > > > > > > > > > > > > > > /python-adapter/src/test/scala/org/apache/sedona/python/wrapper/PythonTestSpec.scala > > > > > > > > > > > > > > /python-adapter/src/test/scala/org/apache/sedona/python/wrapper/GeometrySample.scala > > > > 4. > > > > > > > > > > > > > > /docs/api/javadoc/core/org/apache/sedona/core/geometryObjects/GeoJSONWriterNew.html > > > > /docs/api/javadoc/* > > > > are these doc under /docs/api/javadoc auto generated? if that is the > > > case, > > > > we probably shouldn't have the HTML committed in the repo > > > > 5. > > > > /trunk/import/* > > > > what is this for? > > > > 6. should add a small section on how to build (or link to) the > project > > in > > > > README.md > > > > > > > > > > > > On Wed, Dec 30, 2020 at 6:24 PM Felix Cheung > > > > > wrote: > > > > > > > > > Yes > > > > > https://incubator.apache.org/guides/distribution.html#pypi > > > > > > > > > > Reminder: This should not be posted until we have Sedona and IPMC > > vote. > > > >
Re: Sedona 1.0 Maven release is staged
Hi Jia, I tried to build the Demo project and I have a dependency issue. I tried using sbt assembly and from IntelliJ. For example, build error for Sedona SQL (None of them succeeded in build): https://pastebin.pl/view/15882089 It looks like a bug in SBT regards to SNAPSHOT jars that don't handle: https://github.com/sbt/sbt/issues/3404#issuecomment-400641680 and also here: https://github.com/sbt/sbt/issues/2687 There is any special reason why did you choose to use sbt instead of maven? On Thu, 31 Dec 2020 at 23:27, Jia Yu wrote: > Dear Felix, > > Thanks for your great comments! > > I have fixed all three comments accordingly, Since we have disclaim-WIP, I > think we don't have to re-stage the releases. > > Thanks, > Jia > > > > On Wed, Dec 30, 2020 at 10:54 PM Felix Cheung > wrote: > > > Release review. > > > > Note: with DISCLAIMER-WIP, these are not blockers to release. > > > > website: > > 1. the website can add more links as described in this section: > > https://www.apache.org/foundation/marks/pmcs#navigation > > 2. might need to add trademark attributions > > https://www.apache.org/foundation/marks/pmcs#attributions > > > > project source: > > 1. For LICENSE, we should add RAT check > > https://creadur.apache.org/rat/#What_Is_Rat > > 2. This file seems to have an irregular header: > > /incubator-sedona/zeppelin/index.js > > 3. These code file should have Apache v2 header: > > /python/build_wheel.sh > > /spark-version-converter.py > > > > > /python-adapter/src/test/scala/org/apache/sedona/python/wrapper/PythonTestSpec.scala > > > > > /python-adapter/src/test/scala/org/apache/sedona/python/wrapper/GeometrySample.scala > > 4. > > > > > /docs/api/javadoc/core/org/apache/sedona/core/geometryObjects/GeoJSONWriterNew.html > > /docs/api/javadoc/* > > are these doc under /docs/api/javadoc auto generated? if that is the > case, > > we probably shouldn't have the HTML committed in the repo > > 5. > > /trunk/import/* > > what is this for? > > 6. should add a small section on how to build (or link to) the project in > > README.md > > > > > > On Wed, Dec 30, 2020 at 6:24 PM Felix Cheung > > wrote: > > > > > Yes > > > https://incubator.apache.org/guides/distribution.html#pypi > > > > > > Reminder: This should not be posted until we have Sedona and IPMC vote. > > > > > > > > > On Wed, Dec 30, 2020 at 3:30 PM Jia Yu wrote: > > > > > >> Hi Pawel, > > >> > > >> The only Pypi guideline that I can find is Incubator distribution > > >> guideline: https://incubator.apache.org/guides/distribution.html > > >> > > >> If you scroll down to the bottom, you will see the section. > > >> > > >> @Felix Cheung Felix, does Pawel need to be > > >> aware of anything else, besides the incubator PyPi guideline? > > >> > > >> Thank you very much! > > >> Jia > > >> > > >> On Wed, Dec 30, 2020 at 3:26 PM Jia Yu wrote: > > >> > > >> > Hi Netanel, > > >> > > > >> > I have created a few demo projects: > > >> > https://sedona.staged.apache.org/tutorial/GeoSpark-Runnable-DEMO/ > > >> > The source code is here: > > >> > https://github.com/apache/incubator-sedona/tree/master/examples > > >> > > > >> > Does it look good? > > >> > > > >> > Thanks, > > >> > Jia > > >> > > > >> > On Wed, Dec 30, 2020 at 1:06 PM Netanel Malka > > >> wrote: > > >> > > > >> >> Hi Jia, > > >> >> That's great news! > > >> >> > > >> >> Happy and healthy New Year to everyone. > > >> >> > > >> >> They look fine to me. > > >> >> Do you think it will be worth to use them within a demo project? > (to > > >> >> eliminate dependency issues) I can test it if we need it. > > >> >> > > >> >> BR, > > >> >> > > >> >> > > >> >> Netanel Malka, > > >> >> Big Data Consultant > > >> >> [image: Description: Description: Description: Description: > > >> >> cid:image001.jpg@01C85203.36A2AF30] > > >> >> -- > > >> >> *From:* Paweł Kociński > > >> >> *Sent:* Wednesda
Re: Sedona 1.0 Maven release is staged
Hi Jia, That's great news! Happy and healthy New Year to everyone. They look fine to me. Do you think it will be worth to use them within a demo project? (to eliminate dependency issues) I can test it if we need it. BR, Netanel Malka, Big Data Consultant [Description: Description: Description: Description: cid:image001.jpg@01C85203.36A2AF30] From: Paweł Kociński Sent: Wednesday, December 30, 2020 22:09 To: Jia Yu Cc: dev@sedona.apache.org; Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Zongsi Zhang; Felix Cheung; Netanel Malka Subject: Re: Sedona 1.0 Maven release is staged Hi Jia, Merry Christmas and Happy New Year to everyone. I can definitely do that but I am not sure about the steps which I should follow to upload package under Apache project. Also should we sync this processes, I mean Maven and PyPI release ? Regards, Pawel śr., 30 gru 2020 o 08:43 Jia Yu mailto:ji...@apache.org>> napisał(a): Dear all, Merry Christmas and Happy New Year! After months of hard work, we are finally close to the Sedona 1.0.0-incubator release. I have staged Sedona 1.0.0 official release on Nexus. This is a thread for collecting final feedback rather than a voting thread. 1. I would like to ask all of you to take a look at the staged Maven releases (12 Sedona modules in total). Please learn how to access Nexus from [1][2] @Mohamed Sarwat<mailto:msar...@asu.edu> @Netanel Malka<mailto:netanel...@gmail.com> Would you please take a look at the staged releases? @Felix Cheung<mailto:felixche...@apache.org> Felix, could you please take a look at the release as well? Your opinion is very important. If you think everything is good, then I will call for a vote. 2. Python release on PyPi @Paweł Kociński<mailto:pawel93kocin...@gmail.com> Pawel, could you please publish Sedona 1.0.0-incubator-SNAPSHOT to PyPi? Then we will use that to call for a vote. I know many of you may be on vacation so no need to rush. [1] https://central.sonatype.org/pages/releasing-the-deployment.html#locate-and-examine-your-staging-repository [2] Sedona staged website: https://sedona.staged.apache.org/
Re: First Sedona release
Succeeded to push the snapshots. On Mon, 21 Dec 2020 at 12:04, Netanel Malka wrote: > Thanks. but unfortunately, it's not working. > I got the prompt for the PGP passphrase at the release:prepare phase. > > It looks like I don't have permission to push to the Sedona > nexus artifactory. > > I will try to fix that later. > > On Mon, 21 Dec 2020 at 11:55, Jia Yu wrote: > >> @Netanel Malka >> >> Sometimes, if you are using Mac, you need to enter the following in your >> terminal before using GPG key to sign an artifact: >> https://gist.github.com/jiayuasu/8bab8ecb0234dfc280264fb587fd8b01 >> >> GPG_TTY=$(tty) >> export GPG_TTY >> >> >> >> On Mon, Dec 21, 2020 at 1:52 AM Netanel Malka >> wrote: >> >>> Hi Jia, >>> I tried to deploy but I got a 401 Unauthorized error, full error: >>> https://gist.github.com/netanel246/04c5be423d242a3bb9ef9a300c8817c8 >>> >>> I created a settings.xml file with my apache user and an encrypted >>> password. I also have a GPG key. >>> Did you encounter this problem? >>> >>> >>> Thanks, >>> Netanel Malka. >>> >>> >>> On Sun, 20 Dec 2020 at 20:12, Netanel Malka >>> wrote: >>> >>> > That's great!! >>> > Hope to try it today. >>> > >>> > >>> > On Fri, 18 Dec 2020 at 10:36, Jia Yu wrote: >>> > >>> >> Hi Netanel and Paweł, >>> >> >>> >> The JTS issue has resolved. I am now waiting for JTS 1.18 release but >>> we >>> >> are currently using 1.17.1 + copied files. So we are good anyway. >>> >> >>> >> So the next step will be documentation and stage the first release. >>> >> Although I really want to resolve the ST_Transform lock contention >>> issue, >>> >> it requires a new ST_FlipCoordinate which may take a few days. I will >>> see >>> >> whether I can finish this by Christmas but not sure. >>> >> >>> >> @Netanel Malka Could you please compile the >>> master >>> >> branch and try to deploy a SNAPSHOT release on your own? I have >>> pushed a >>> >> few snapshots but I would like to see whether you can do it too. >>> Please >>> >> follow the steps here: >>> >> https://gist.github.com/jiayuasu/849e1f3bf7a2dd11593ca27c14e9e92d >>> >> >>> >> @Paweł Kociński Step 1. Could you please >>> >> update >>> >> the new Python Adaptor documentation? Step 2. Could you please try to >>> >> deploy a SNAPSHOT release to PyPI? You can find some help here: >>> >> https://incubator.apache.org/guides/distribution.html >>> >> >>> >> Thank you very much! >>> >> Jia >>> >> >>> >> >>> >> On Thu, Dec 10, 2020 at 3:26 PM Jim Hughes wrote: >>> >> >>> >> > Hi Jia, >>> >> > >>> >> > A JTS 1.18.0 release would not be just for Apache Sedona.;) Getting >>> it >>> >> > out sooner would let others projects adopt it sooner (I'm thinking >>> of >>> >> > GeoTools and GeoServer). I'm excited to see the improvements to the >>> >> > overlay operations... >>> >> > >>> >> > I've traded some emails and chats with Martin. It sounds like he >>> is ok >>> >> > with cutting JTS 1.18.0 in the next week; I'll be working with him >>> and >>> >> > Jody to do our best to make that happen. >>> >> > >>> >> > Anyhow, in terms of shading, there are few things I'd suggest. >>> First, >>> >> > I'd suggest that libraries which can function as libraries have a >>> >> > version of the jar which does not include any dependencies. If you >>> go >>> >> > along with that, sedona-core should produce a jar on its own and >>> another >>> >> > module could build a "batteries included" jar for users to drop into >>> >> Spark. >>> >> > >>> >> > Separate from that, I'd recommend that when you copy entire files >>> into a >>> >> > project that you change the package for those classes. Concretely, >>> you >>> >> > could just prepend org.apache.sedona to the package names for those >>> 5 >&g
Re: First Sedona release
Thanks. but unfortunately, it's not working. I got the prompt for the PGP passphrase at the release:prepare phase. It looks like I don't have permission to push to the Sedona nexus artifactory. I will try to fix that later. On Mon, 21 Dec 2020 at 11:55, Jia Yu wrote: > @Netanel Malka > > Sometimes, if you are using Mac, you need to enter the following in your > terminal before using GPG key to sign an artifact: > https://gist.github.com/jiayuasu/8bab8ecb0234dfc280264fb587fd8b01 > > GPG_TTY=$(tty) > export GPG_TTY > > > > On Mon, Dec 21, 2020 at 1:52 AM Netanel Malka > wrote: > >> Hi Jia, >> I tried to deploy but I got a 401 Unauthorized error, full error: >> https://gist.github.com/netanel246/04c5be423d242a3bb9ef9a300c8817c8 >> >> I created a settings.xml file with my apache user and an encrypted >> password. I also have a GPG key. >> Did you encounter this problem? >> >> >> Thanks, >> Netanel Malka. >> >> >> On Sun, 20 Dec 2020 at 20:12, Netanel Malka wrote: >> >> > That's great!! >> > Hope to try it today. >> > >> > >> > On Fri, 18 Dec 2020 at 10:36, Jia Yu wrote: >> > >> >> Hi Netanel and Paweł, >> >> >> >> The JTS issue has resolved. I am now waiting for JTS 1.18 release but >> we >> >> are currently using 1.17.1 + copied files. So we are good anyway. >> >> >> >> So the next step will be documentation and stage the first release. >> >> Although I really want to resolve the ST_Transform lock contention >> issue, >> >> it requires a new ST_FlipCoordinate which may take a few days. I will >> see >> >> whether I can finish this by Christmas but not sure. >> >> >> >> @Netanel Malka Could you please compile the >> master >> >> branch and try to deploy a SNAPSHOT release on your own? I have pushed >> a >> >> few snapshots but I would like to see whether you can do it too. Please >> >> follow the steps here: >> >> https://gist.github.com/jiayuasu/849e1f3bf7a2dd11593ca27c14e9e92d >> >> >> >> @Paweł Kociński Step 1. Could you please >> >> update >> >> the new Python Adaptor documentation? Step 2. Could you please try to >> >> deploy a SNAPSHOT release to PyPI? You can find some help here: >> >> https://incubator.apache.org/guides/distribution.html >> >> >> >> Thank you very much! >> >> Jia >> >> >> >> >> >> On Thu, Dec 10, 2020 at 3:26 PM Jim Hughes wrote: >> >> >> >> > Hi Jia, >> >> > >> >> > A JTS 1.18.0 release would not be just for Apache Sedona.;) Getting >> it >> >> > out sooner would let others projects adopt it sooner (I'm thinking of >> >> > GeoTools and GeoServer). I'm excited to see the improvements to the >> >> > overlay operations... >> >> > >> >> > I've traded some emails and chats with Martin. It sounds like he is >> ok >> >> > with cutting JTS 1.18.0 in the next week; I'll be working with him >> and >> >> > Jody to do our best to make that happen. >> >> > >> >> > Anyhow, in terms of shading, there are few things I'd suggest. First, >> >> > I'd suggest that libraries which can function as libraries have a >> >> > version of the jar which does not include any dependencies. If you >> go >> >> > along with that, sedona-core should produce a jar on its own and >> another >> >> > module could build a "batteries included" jar for users to drop into >> >> Spark. >> >> > >> >> > Separate from that, I'd recommend that when you copy entire files >> into a >> >> > project that you change the package for those classes. Concretely, >> you >> >> > could just prepend org.apache.sedona to the package names for those 5 >> >> > classes. (This assumes that it is possible. Sometimes there may be >> >> > issues around package protected access, etc.) >> >> > >> >> > As it stands right now, if a user tries to use Sedona with any other >> >> > library that pulls in JTS, then they will be at the mercy of the >> class >> >> > loading order. If the JTS jar comes in elsewhere, your versions of >> the >> >> > RTree may not be loaded! The exception would look like a JTS issue >> and &g
Re: First Sedona release
Hi Jia, I tried to deploy but I got a 401 Unauthorized error, full error: https://gist.github.com/netanel246/04c5be423d242a3bb9ef9a300c8817c8 I created a settings.xml file with my apache user and an encrypted password. I also have a GPG key. Did you encounter this problem? Thanks, Netanel Malka. On Sun, 20 Dec 2020 at 20:12, Netanel Malka wrote: > That's great!! > Hope to try it today. > > > On Fri, 18 Dec 2020 at 10:36, Jia Yu wrote: > >> Hi Netanel and Paweł, >> >> The JTS issue has resolved. I am now waiting for JTS 1.18 release but we >> are currently using 1.17.1 + copied files. So we are good anyway. >> >> So the next step will be documentation and stage the first release. >> Although I really want to resolve the ST_Transform lock contention issue, >> it requires a new ST_FlipCoordinate which may take a few days. I will see >> whether I can finish this by Christmas but not sure. >> >> @Netanel Malka Could you please compile the master >> branch and try to deploy a SNAPSHOT release on your own? I have pushed a >> few snapshots but I would like to see whether you can do it too. Please >> follow the steps here: >> https://gist.github.com/jiayuasu/849e1f3bf7a2dd11593ca27c14e9e92d >> >> @Paweł Kociński Step 1. Could you please >> update >> the new Python Adaptor documentation? Step 2. Could you please try to >> deploy a SNAPSHOT release to PyPI? You can find some help here: >> https://incubator.apache.org/guides/distribution.html >> >> Thank you very much! >> Jia >> >> >> On Thu, Dec 10, 2020 at 3:26 PM Jim Hughes wrote: >> >> > Hi Jia, >> > >> > A JTS 1.18.0 release would not be just for Apache Sedona.;) Getting it >> > out sooner would let others projects adopt it sooner (I'm thinking of >> > GeoTools and GeoServer). I'm excited to see the improvements to the >> > overlay operations... >> > >> > I've traded some emails and chats with Martin. It sounds like he is ok >> > with cutting JTS 1.18.0 in the next week; I'll be working with him and >> > Jody to do our best to make that happen. >> > >> > Anyhow, in terms of shading, there are few things I'd suggest. First, >> > I'd suggest that libraries which can function as libraries have a >> > version of the jar which does not include any dependencies. If you go >> > along with that, sedona-core should produce a jar on its own and another >> > module could build a "batteries included" jar for users to drop into >> Spark. >> > >> > Separate from that, I'd recommend that when you copy entire files into a >> > project that you change the package for those classes. Concretely, you >> > could just prepend org.apache.sedona to the package names for those 5 >> > classes. (This assumes that it is possible. Sometimes there may be >> > issues around package protected access, etc.) >> > >> > As it stands right now, if a user tries to use Sedona with any other >> > library that pulls in JTS, then they will be at the mercy of the class >> > loading order. If the JTS jar comes in elsewhere, your versions of the >> > RTree may not be loaded! The exception would look like a JTS issue and >> > it be fairly confusing for most people to debug. >> > >> > With those issues taken together, a user could load up a sedona-core jar >> > (which wouldn't have JTS or org.wololo.geojson) with a different version >> > of JTS potentially provided by another project and be able to use Sedona >> > and other projects together. >> > >> > Thanks for working through the issues to be able to use a release of >> > JTS. Hopefully we can knock this out over the next week, and if not, >> > you do have an approach which would let you release Sedona. >> > >> > Cheers, >> > >> > Jim >> > >> > On 12/10/2020 2:33 PM, Jia Yu wrote: >> > > Hi Jim, >> > > >> > > Thanks for your feedback. >> > > >> > > 1. I indeed asked Martin, Jody, and you in the JTS Gitter chat. It >> looks >> > > like Martin still needs some time to fix some functions. In fact, I >> feel >> > it >> > > is inappropriate to push Martin, an OSS contributor, to draw a release >> > just >> > > for us :) >> > > 2. I also saw your comment on the GitHub PR. My current solution in >> that >> > PR >> > > is that use JTS 1.17.1 official release + 5 copied JTS index classes. >> I
Re: First Sedona release
That's great!! Hope to try it today. On Fri, 18 Dec 2020 at 10:36, Jia Yu wrote: > Hi Netanel and Paweł, > > The JTS issue has resolved. I am now waiting for JTS 1.18 release but we > are currently using 1.17.1 + copied files. So we are good anyway. > > So the next step will be documentation and stage the first release. > Although I really want to resolve the ST_Transform lock contention issue, > it requires a new ST_FlipCoordinate which may take a few days. I will see > whether I can finish this by Christmas but not sure. > > @Netanel Malka Could you please compile the master > branch and try to deploy a SNAPSHOT release on your own? I have pushed a > few snapshots but I would like to see whether you can do it too. Please > follow the steps here: > https://gist.github.com/jiayuasu/849e1f3bf7a2dd11593ca27c14e9e92d > > @Paweł Kociński Step 1. Could you please > update > the new Python Adaptor documentation? Step 2. Could you please try to > deploy a SNAPSHOT release to PyPI? You can find some help here: > https://incubator.apache.org/guides/distribution.html > > Thank you very much! > Jia > > > On Thu, Dec 10, 2020 at 3:26 PM Jim Hughes wrote: > > > Hi Jia, > > > > A JTS 1.18.0 release would not be just for Apache Sedona.;) Getting it > > out sooner would let others projects adopt it sooner (I'm thinking of > > GeoTools and GeoServer). I'm excited to see the improvements to the > > overlay operations... > > > > I've traded some emails and chats with Martin. It sounds like he is ok > > with cutting JTS 1.18.0 in the next week; I'll be working with him and > > Jody to do our best to make that happen. > > > > Anyhow, in terms of shading, there are few things I'd suggest. First, > > I'd suggest that libraries which can function as libraries have a > > version of the jar which does not include any dependencies. If you go > > along with that, sedona-core should produce a jar on its own and another > > module could build a "batteries included" jar for users to drop into > Spark. > > > > Separate from that, I'd recommend that when you copy entire files into a > > project that you change the package for those classes. Concretely, you > > could just prepend org.apache.sedona to the package names for those 5 > > classes. (This assumes that it is possible. Sometimes there may be > > issues around package protected access, etc.) > > > > As it stands right now, if a user tries to use Sedona with any other > > library that pulls in JTS, then they will be at the mercy of the class > > loading order. If the JTS jar comes in elsewhere, your versions of the > > RTree may not be loaded! The exception would look like a JTS issue and > > it be fairly confusing for most people to debug. > > > > With those issues taken together, a user could load up a sedona-core jar > > (which wouldn't have JTS or org.wololo.geojson) with a different version > > of JTS potentially provided by another project and be able to use Sedona > > and other projects together. > > > > Thanks for working through the issues to be able to use a release of > > JTS. Hopefully we can knock this out over the next week, and if not, > > you do have an approach which would let you release Sedona. > > > > Cheers, > > > > Jim > > > > On 12/10/2020 2:33 PM, Jia Yu wrote: > > > Hi Jim, > > > > > > Thanks for your feedback. > > > > > > 1. I indeed asked Martin, Jody, and you in the JTS Gitter chat. It > looks > > > like Martin still needs some time to fix some functions. In fact, I > feel > > it > > > is inappropriate to push Martin, an OSS contributor, to draw a release > > just > > > for us :) > > > 2. I also saw your comment on the GitHub PR. My current solution in > that > > PR > > > is that use JTS 1.17.1 official release + 5 copied JTS index classes. I > > > also use the maven shade plugin to filter out the 5 corresponding > classes > > > in JTS 1.17.1 jar ( > > > > > > https://github.com/apache/incubator-sedona/pull/495/files#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8R278 > > ) > > > to avoid duplicates . Do you think I should even use the shade plugin > to > > > relocate these classes to a different path? > > > > > > Thanks, > > > Jia > > > > > > On Thu, Dec 10, 2020 at 6:25 AM Jim Hughes wrote: > > > > > >> Hi all, > > >> > > >> It may be worth discussing with the JTS directly what their schedule > i
Re: First Sedona release
then work on this one? If >> this >> >>>>>> Python RDD-DF Adapter PR might slow down our progress of releasing >> >>>> Sedona >> >>>>>> before Christmas, we can postpone it to Sedona 1.0.1 or 1.1.0. >> >>>>>> >> >>>>>> @everyone >> >>>>>> Our top priority is to draw the first Sedona release ASAP. Users >> have >> >>>> been >> >>>>>> waiting for almost six months. Let's push hard to publish the first >> >>>> Sedona >> >>>>>> release to Maven Central and PyPI before Christmas. In order to >> make >> >> it >> >>>>>> happen, >> >>>>>> >> >>>>>> Finalize coding and documentation before Dec 6: >> >>>>>> 1. I believe the Move-to-JTS PR will be done in around one week. >> >>>>>> 2. Then we can accept Pawel' Python RDD-DF Adapter PR, if necessary >> >>>>>> 3. I will work on Sedona documentation. >> >>>>>> 4. @Netanel will work on Sedona support of Spark 2.4 and Scala >> 2.11. >> >> I >> >>>> will >> >>>>>> first create a branch for it to illustrate some necessary changes >> in >> >>>> Sedona >> >>>>>> SQL for Spark 2.4. >> >>>>>> >> >>>>>> Final walk-through before Dec 13 >> >>>>>> 1. Netanel can test the release management for Sedona. >> >>>>>> 2. Other committers can go through the docs, release notes >> >>>>>> >> >>>>>> Community voting before Dec 20 >> >>>>>> 1. Sedona community voting: before Dec 16 >> >>>>>> 2. Apache Incubator voting: before Dec 20 >> >>>>>> >> >>>>>> Push to Maven Central and PyPi before Dec 24 >> >>>>>> >> >>>>>> Please feel free to comment if you have any suggestions! >> >>>>>> >> >>>>>> Jia >> >>>>>> >> >>>>>> On Sun, Nov 22, 2020 at 9:51 AM Paweł Kociński < >> >>>> pawel93kocin...@gmail.com> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> Hi, >> >>>>>>> I saw some users reported need to improve Python RDD API in two >> >>>>>> scenarios: >> >>>>>>> - converting spatial flat join result to df >> >>>>>>> - saving spatial flat join result directly to external storage >> >>>>>>> >> >>>>>>> Currently SerDe between jvm and Python causes additional time >> needed >> >>>> to >> >>>>>>> compute the result. I have a local branch with tests where this >> >>>>>>> functionality is available (need 3-4 days to make it 100% ready), >> in >> >>>> two >> >>>>>>> above scenarios there will be almost no difference between Python >> >> and >> >>>>>> Scala >> >>>>>>> or Java API. Should I create PR to include this feature within the >> >>>> first >> >>>>>>> Sedona release ? >> >>>>>>> Regards, >> >>>>>>> Paweł >> >>>>>>> >> >>>>>>> pon., 16 lis 2020 o 08:29 Jia Yu >> >> napisał(a): >> >>>>>>>> Dear all, >> >>>>>>>> >> >>>>>>>> Thanks for all your suggestions. >> >>>>>>>> >> >>>>>>>> 1. To completely solve the long-overdue JTS issue, I made a >> Sedona >> >> PR >> >>>>>> and >> >>>>>>>> two JTS PRs. @Jim Hughes , @Paweł Kociński >> >>>>>>>> , I, and probably Martin from JTS >> will >> >>>> take >> >>>>>>>> care of these PRs in the coming days. >> >>>>>>>> (1) Sedona PR: >> https://github.com/apache/incubator-sedona/pull/488 >> >>>>>>>> (2) JTS PR: https://github.com/locationtech/jts/pull/633 >> >>>>>>>> https://github.com/loca
JTS 1.16.2
Hi Jia, I tried to use the branch sedona-1.0-doc but the artifact for 1.16.2 is not in the Maven Central so I cannot compile the project. I got: The POM for org.locationtech.jts:jts-core:jar:1.16.2 is missing, no dependency information available I searched for the artifact here: https://mvnrepository.com/artifact/org.locationtech.jts/jts-core And also in the JCenter repo here: https://bintray.com/bintray/jcenter/org.locationtech.jts%3Ajts-core#files/org/locationtech/jts/jts-core As you can see there is no artifact for 1.16.2 only for 1.16.1 or 1.17.x Can you downgrade to 1.16.1 or upgrade to 1.17.1? Thanks.
Re: Support Spark 2.4 in Sedona 1.0
What do you mean by compile target? ? Iv'e found Apache Zeppelin handle multiple Spark versions here using profile for each Spark version: https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L185 Do you think this method is better? Netanel Malka, Big Data Consultant [Description: Description: Description: Description: cid:image001.jpg@01C85203.36A2AF30] From: Felix Cheung Sent: Thursday, November 12, 2020 04:05 To: dev@sedona.apache.org Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi Zhang; d...@sedona.incubator.apache.org Subject: Re: Support Spark 2.4 in Sedona 1.0 I am not sure it should be a branch? It is common to deal with this as a compile target, not as a separate branch. A separate branch might have difficulty to release? There are a few example in projects where they handle multiple Spark target version like this. On Wed, Nov 11, 2020 at 12:56 PM Jia Yu mailto:ji...@apache.org>> wrote: OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the compiler used in each branch, For Sedona on Spark 3.0, I will compile it using Scala 2.12 For Sedona on Spark 2.4, I will compile it using Scala 2.11. For the Java code in both branches, I will compile them using Java 1.8 Am I missing anything here? On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka mailto:netan...@sela.co.il>> wrote: > Hi, > > I also think that we need to support 2.4. > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep > 12, 2020) > > I also asked about it on > *us...@spark.apache.org<mailto:us...@spark.apache.org> > mailto:us...@spark.apache.org>>* > : > > > *Sean Owen (answered the question): * > > "I don't think there's an official EOL for Spark 2.4.x but would expect > another maintenance release in the first half of 2021 at least. I'd also > guess it wouldn't be maintained by 2022." > > > ?BR, > > > > Netanel Malka, > Big Data Consultant > [image: Description: Description: Description: Description: > cid:image001.jpg@01C85203.36A2AF30] > -- > *From:* Paweł Kociński > mailto:pawel93kocin...@gmail.com>> > *Sent:* Wednesday, November 11, 2020 00:29 > *To:* Jia Yu > *Cc:* dev@sedona.apache.org<mailto:dev@sedona.apache.org>; > d...@sedona.incubator.apache.org<mailto:d...@sedona.incubator.apache.org>; > Jinxuan Wu; > Mohamed Sarwat; Netanel Malka; Zongsi Zhang > *Subject:* Re: Support Spark 2.4 in Sedona 1.0 > > Hi Jia, > I think we should support spark 2.4, a lot of users still use it. More > than that I think more users still have jobs written in spark 2.4 than > 3.0. We will use an additional branch for that use case ? I mean Spark 2.4 > with scala 2.12 is important one. > Regards, > Paweł > > pon., 9 lis 2020 o 20:44 Jia Yu mailto:ji...@apache.org>> > napisał(a): > >> Dear all, >> >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder whether >> we should support Spark 2.4. >> >> In order to support Spark 2.4, we need to do the following >> >> 1. Compile the source using Scala 2.11. Sedona master branch currently is >> compiled by Scala 2.12 and Java 1.8 >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the (1) >> UDF registration hook (2) the SQL aggregation function format >> 3. In the future releases of Sedona, use git cherry-pick to pick >> important features back to the Spark 2.4 branch. This is what I did in >> GeoSpark to support Spark 2.1, 2.2, 2.3 >> >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it >> that way and just support Spark 3.0. >> >> Do you think we should support Spark 2.4 in the future release? >> >> Thanks, >> Jia Yu >> >
Re: Support Spark 2.4 in Sedona 1.0
Hi, I also think that we need to support 2.4. I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep 12, 2020) I also asked about it on us...@spark.apache.org : Sean Owen (answered the question): "I don't think there's an official EOL for Spark 2.4.x but would expect another maintenance release in the first half of 2021 at least. I'd also guess it wouldn't be maintained by 2022." ?BR, Netanel Malka, Big Data Consultant [Description: Description: Description: Description: cid:image001.jpg@01C85203.36A2AF30] From: Paweł Kociński Sent: Wednesday, November 11, 2020 00:29 To: Jia Yu Cc: dev@sedona.apache.org; d...@sedona.incubator.apache.org; Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Zongsi Zhang Subject: Re: Support Spark 2.4 in Sedona 1.0 Hi Jia, I think we should support spark 2.4, a lot of users still use it. More than that I think more users still have jobs written in spark 2.4 than 3.0. We will use an additional branch for that use case ? I mean Spark 2.4 with scala 2.12 is important one. Regards, Paweł pon., 9 lis 2020 o 20:44 Jia Yu mailto:ji...@apache.org>> napisał(a): Dear all, In Sedona 1.0, we definitely will support Spark 3.0. But I wonder whether we should support Spark 2.4. In order to support Spark 2.4, we need to do the following 1. Compile the source using Scala 2.11. Sedona master branch currently is compiled by Scala 2.12 and Java 1.8 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the (1) UDF registration hook (2) the SQL aggregation function format 3. In the future releases of Sedona, use git cherry-pick to pick important features back to the Spark 2.4 branch. This is what I did in GeoSpark to support Spark 2.1, 2.2, 2.3 GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it that way and just support Spark 3.0. Do you think we should support Spark 2.4 in the future release? Thanks, Jia Yu
Re: First Sedona release
OK. Thanks Felix. Updates: * * Opened a ticket for INFRA to Enable Nexus Access For Sedona<https://issues.apache.org/jira/browse/INFRA-21085> * Followed this<https://infra.apache.org/publishing-maven-artifacts.html> guide to test the maven release process * I hope to create a PR soon for adjusting the build to deploy to the ASF Nexus repository * The key that signs the artifacts were created and tested. Do we want to create a candidate release for the current master branch? Netanel Malka, Big Data Consultant [Description: Description: Description: Description: cid:image001.jpg@01C85203.36A2AF30] From: Felix Cheung Sent: Wednesday, November 4, 2020 19:57 To: dev@sedona.apache.org Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi Zhang Subject: Re: First Sedona release 1) No you don’t need KEYS file in github only on the release share https://dist.apache.org/repos/dist/dev/incubator/ 2) as podling you add to https://dist.apache.org/repos/dist/dev/incubator/ When you commit via svn you will be able to add a “directory” for Sedona 2a) for release, you basically do a svn rename to move from dev to release “path” 3) if you have java based artifacts, yes. You will publish to Nexus, staging first and when release is signed off, you can click on the interface to make it official, which then automatically sync to Maven central. Here is a script for example that does release signing and publication to Nexus (and staging before release) https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh On Wed, Nov 4, 2020 at 2:50 AM Netanel Malka mailto:netanel...@gmail.com>> wrote: Hi, I followed the release-signing <https://infra.apache.org/release-signing.html> doc and created a key for signing and hashing. I have a few questions: 1. Should the KEYS file also be added to the project root directory on Github? ( I saw it in Apache Ant) 2. I saw in release-policy_upload-ci <http://www.apache.org/legal/release-policy.html#upload-ci> that we need to add a release candidate to https://dist.apache.org/repos/dist/*dev*//. However, there does not seem to be a directory with Sedona as the TLP name. How may we be able to get a directory with that name? (Also for the *release*) 3. Do we need to push the artifacts also to ASF Nexus Repository (beside Maven Central)? Thanks. On Mon, 2 Nov 2020 at 19:21, Netanel Malka mailto:netanel...@gmail.com>> wrote: > Thanks Felix. > > I would be delighted to help. > I can start with the GPG. > Can I test it on a some artifact, or I need to wait for the first release? > > > On Mon, 2 Nov 2020 at 03:17, Felix Cheung > mailto:felixche...@apache.org>> wrote: > >> Great progress! >> >> To add, >> A) I’d strongly recommend the WIP disclaimer - it would be much easier to >> pass with in the first release >> https://incubator.apache.org/policy/incubation.html#disclaimers >> >> B) more info in signing, checksum >> https://infra.apache.org/release-signing.html >> >> C) signing key should be individual’s and (public key ) published and also >> listed in KEYS file - KEYS file should be located next to the staging >> (and >> later release) location, see above >> >> D) “correct place” - this is in reference to ASF officIal staging server >> http://www.apache.org/legal/release-policy.html#stage >> And can be “uploaded” by committing to svn >> http://www.apache.org/legal/release-policy.html#upload-ci >> >> E) python / PyPI - >> https://incubator.apache.org/guides/distribution.html#pypi >> >> >> >> On Sun, Nov 1, 2020 at 2:17 PM Jia Yu >> mailto:ji...@apache.org>> wrote: >> >> > Hi Netanel, Pawel and other committers, >> > >> > While Pawel is working on Python code of Sedona 1.0, let's focus on >> other >> > parts required by the release. Netanel, can you help me with all the ASF >> > incubator requirement items that are not DONE? >> > >> > *Here is a checklist for our first Sedona release* >> > >> > *ASF incubator requirement >> > (https://incubator.apache.org/guides/releasemanagement.html >> > <https://incubator.apache.org/guides/releasemanagement.html>, we >> probably >> > should read ASF release requirement as well):* >> > >> > 1 .Include the word incubating in the release file name: DONE. Please >> see >> > the POM.xml in all directories. >> > >> > 2. Include an ASF LICENSE and NOTICE file: DONE. Please see the GitHub >> > repo. >> > >> > 3. Have valid checksums or signatures: I believe signature should be >>
Re: First Sedona release
Hi, I followed the release-signing <https://infra.apache.org/release-signing.html> doc and created a key for signing and hashing. I have a few questions: 1. Should the KEYS file also be added to the project root directory on Github? ( I saw it in Apache Ant) 2. I saw in release-policy_upload-ci <http://www.apache.org/legal/release-policy.html#upload-ci> that we need to add a release candidate to https://dist.apache.org/repos/dist/*dev*//. However, there does not seem to be a directory with Sedona as the TLP name. How may we be able to get a directory with that name? (Also for the *release*) 3. Do we need to push the artifacts also to ASF Nexus Repository (beside Maven Central)? Thanks. On Mon, 2 Nov 2020 at 19:21, Netanel Malka wrote: > Thanks Felix. > > I would be delighted to help. > I can start with the GPG. > Can I test it on a some artifact, or I need to wait for the first release? > > > On Mon, 2 Nov 2020 at 03:17, Felix Cheung wrote: > >> Great progress! >> >> To add, >> A) I’d strongly recommend the WIP disclaimer - it would be much easier to >> pass with in the first release >> https://incubator.apache.org/policy/incubation.html#disclaimers >> >> B) more info in signing, checksum >> https://infra.apache.org/release-signing.html >> >> C) signing key should be individual’s and (public key ) published and also >> listed in KEYS file - KEYS file should be located next to the staging >> (and >> later release) location, see above >> >> D) “correct place” - this is in reference to ASF officIal staging server >> http://www.apache.org/legal/release-policy.html#stage >> And can be “uploaded” by committing to svn >> http://www.apache.org/legal/release-policy.html#upload-ci >> >> E) python / PyPI - >> https://incubator.apache.org/guides/distribution.html#pypi >> >> >> >> On Sun, Nov 1, 2020 at 2:17 PM Jia Yu wrote: >> >> > Hi Netanel, Pawel and other committers, >> > >> > While Pawel is working on Python code of Sedona 1.0, let's focus on >> other >> > parts required by the release. Netanel, can you help me with all the ASF >> > incubator requirement items that are not DONE? >> > >> > *Here is a checklist for our first Sedona release* >> > >> > *ASF incubator requirement >> > (https://incubator.apache.org/guides/releasemanagement.html >> > <https://incubator.apache.org/guides/releasemanagement.html>, we >> probably >> > should read ASF release requirement as well):* >> > >> > 1 .Include the word incubating in the release file name: DONE. Please >> see >> > the POM.xml in all directories. >> > >> > 2. Include an ASF LICENSE and NOTICE file: DONE. Please see the GitHub >> > repo. >> > >> > 3. Have valid checksums or signatures: I believe signature should be >> done >> > by the GPG key. Not sure about the checksum. I am also not sure about >> the >> > GPG key requirement of ASF. I use GPG key to sign releases of GeoSpark >> in >> > the past. >> > >> > 4. Be placed in the correct place on the ASF’s infrastructure: we should >> > place our releases in two places: Maven, and PyPi. Not sure how to >> relate >> > them to ASF. >> > >> > 5. Have a KEYS file to validate the release: this should be the public >> key >> > of our GPG key? >> > >> > *Sedona requirement* >> > >> > 1. Python path name, file headers, and jars >> > 2. Project website docs: documentation should use the name, Sedona, in >> all >> > tutorials. We should also include the situation of GeoTools >> dependencies. >> > >> > Thanks, >> > Jia >> > >> > >> > On Wed, Oct 14, 2020 at 10:08 PM Jia Yu wrote: >> > >> > > Hi folks, >> > > >> > > We will be working on the first Sedona. Please see the JIRA ticket >> here: >> > > >> > >> https://issues.apache.org/jira/projects/SEDONA/issues/SEDONA-3?filter=allopenissues >> > > >> > > Do you think there are any outstanding issues to be fixed as well? >> > > >> > > Thanks, >> > > Jia >> > > >> > >> > > > -- > Best regards, > Netanel Malka. > -- Best regards, Netanel Malka.
Re: First Sedona release
Thanks Felix. I would be delighted to help. I can start with the GPG. Can I test it on a some artifact, or I need to wait for the first release? On Mon, 2 Nov 2020 at 03:17, Felix Cheung wrote: > Great progress! > > To add, > A) I’d strongly recommend the WIP disclaimer - it would be much easier to > pass with in the first release > https://incubator.apache.org/policy/incubation.html#disclaimers > > B) more info in signing, checksum > https://infra.apache.org/release-signing.html > > C) signing key should be individual’s and (public key ) published and also > listed in KEYS file - KEYS file should be located next to the staging (and > later release) location, see above > > D) “correct place” - this is in reference to ASF officIal staging server > http://www.apache.org/legal/release-policy.html#stage > And can be “uploaded” by committing to svn > http://www.apache.org/legal/release-policy.html#upload-ci > > E) python / PyPI - > https://incubator.apache.org/guides/distribution.html#pypi > > > > On Sun, Nov 1, 2020 at 2:17 PM Jia Yu wrote: > > > Hi Netanel, Pawel and other committers, > > > > While Pawel is working on Python code of Sedona 1.0, let's focus on other > > parts required by the release. Netanel, can you help me with all the ASF > > incubator requirement items that are not DONE? > > > > *Here is a checklist for our first Sedona release* > > > > *ASF incubator requirement > > (https://incubator.apache.org/guides/releasemanagement.html > > <https://incubator.apache.org/guides/releasemanagement.html>, we > probably > > should read ASF release requirement as well):* > > > > 1 .Include the word incubating in the release file name: DONE. Please see > > the POM.xml in all directories. > > > > 2. Include an ASF LICENSE and NOTICE file: DONE. Please see the GitHub > > repo. > > > > 3. Have valid checksums or signatures: I believe signature should be done > > by the GPG key. Not sure about the checksum. I am also not sure about the > > GPG key requirement of ASF. I use GPG key to sign releases of GeoSpark in > > the past. > > > > 4. Be placed in the correct place on the ASF’s infrastructure: we should > > place our releases in two places: Maven, and PyPi. Not sure how to relate > > them to ASF. > > > > 5. Have a KEYS file to validate the release: this should be the public > key > > of our GPG key? > > > > *Sedona requirement* > > > > 1. Python path name, file headers, and jars > > 2. Project website docs: documentation should use the name, Sedona, in > all > > tutorials. We should also include the situation of GeoTools dependencies. > > > > Thanks, > > Jia > > > > > > On Wed, Oct 14, 2020 at 10:08 PM Jia Yu wrote: > > > > > Hi folks, > > > > > > We will be working on the first Sedona. Please see the JIRA ticket > here: > > > > > > https://issues.apache.org/jira/projects/SEDONA/issues/SEDONA-3?filter=allopenissues > > > > > > Do you think there are any outstanding issues to be fixed as well? > > > > > > Thanks, > > > Jia > > > > > > -- Best regards, Netanel Malka.
JTSplus fork status
Hi Jia, I saw that you merge the PR about make Sedona depend on JTS. Did you also try to open a PR in the JTS project? I wondered if they agreed to accept your changes. BR, Netanel Malka.
Re: Apache Sedona
anch) >>> - Add direct writing to geospatial databases like PostgreSQL >>> - Add more geospatial functions >>> - Remove NullPointer exception when there is null value within data or >>> data is wrong within some rows >>> - geohash spatial join >>> >>> What do you think? >>> >>> Regards, >>> Paweł >>> >>> >>> pon., 17 sie 2020 o 07:45 Jia Yu napisał(a): >>> >>>> Hello Paweł, >>>> >>>> I just posted the current situation into priv...@sedona.apache.org. >>>> The current problem is I have made everything ready to be imported to ASF >>>> GitHub repo (https://github.com/apache/incubator-sedona). But one >>>> committer (Masha from Facebook) who made thousands of lines of contribution >>>> to GeoSpark still didn't submit her CLA. The entire process is currently >>>> blocked by this. >>>> >>>> Mohamed and I have been trying to reach her a couple of times in the >>>> past 3 weeks but got no reply. I have asked the champion about how we can >>>> proceed in this case. Let's see what will happen. >>>> >>>> Thanks, >>>> Jia >>>> >>>> >>>> On Sun, Aug 16, 2020 at 9:06 AM Paweł Kociński < >>>> pawel93kocin...@gmail.com> wrote: >>>> >>>>> Hi Jia, >>>>> Do we know when the first release of Apache Sedona will occur ? Can I >>>>> help with something to make it happen? I have few ideas and some code >>>>> which >>>>> will be useful in the future. >>>>> >>>>> Regards, >>>>> Pawel >>>>> >>>> -- Best regards, Netanel Malka.
Re: Use JTS as a dependency instead of JTSPlus
Yes, that's right. We are already working on that. I hope to create the PR soon. On Fri, Aug 21, 2020, 11:35 Jia Yu wrote: > Hi folks, > > I believe the conclusion is that we should use the wrapper solution > instead of the reflection, right? (of course, with additional care to the > wrapper) > > Thanks, > Jia > > On Sun, Aug 9, 2020 at 11:37 PM Paweł Kociński > wrote: > >> Hi, >> From my point of view, Python API needs only a few changes in that case. >> First of all, few type annotation names change (Python API already has some >> proxy object which holds shapely geometry and user data as a >> separate attribute), If the new object has getter *getUserData, *the >> change should be minimal. And those are changes for RDD API. SQL API should >> not require changes due to the fact that translation between Dataframe and >> RDD is hidden for Python (I assume that GeometryUDT will remain the same). >> >> Regards, >> Pawel >> >> pon., 10 sie 2020 o 07:08 Georg Heiler napisał(a): >> >>> I agree with @Jia Yu and think it is better to >>> move forward with the wrapper. >>> >>> Best, >>> Georg >>> >>> Am Mo., 10. Aug. 2020 um 01:41 Uhr schrieb Jia Yu : >>> >>>> Hi Netanel, CCed Pawel (GeoSpark Python), Georg (who might be also >>>> interested in this issue), Sedona-dev >>>> >>>> I think reflection would be a neat solution but it may bring >>>> technical debt in the future and cause problems to the python API. >>>> >>>> In the long run, a wrapper around JTS geometry would be a better >>>> solution although we may need to change many places in the code. >>>> >>>> Folks, what do you think? >>>> >>>> Thanks, >>>> Jia >>>> >>>> On Sun, Aug 9, 2020 at 7:49 AM Netanel Malka >>>> wrote: >>>> >>>>> Hi, >>>>> Currently, we are having some problems with userData on Geometry. >>>>> The problems are: >>>>> >>>>>1. Geometry toString function doesn't take userData into account >>>>>2. Geometry equals function doesn't take userData into account >>>>> >>>>> >>>>> Our proposed solution is to wrap Geometry with a proxy object, that >>>>> holds the Geometry and handles other columns instead of using Goemtery >>>>> user >>>>> data. >>>>> Another possible solution is using reflection to change methods on >>>>> Geometry itself >>>>> >>>>> What do you think we should do? >>>>> >>>>> Thanks. Regards >>>>> >>>>> On Thu, Jul 23, 2020, 21:32 Jia Yu wrote: >>>>> >>>>>> Hi Netanel, >>>>>> >>>>>> Sorry. I somehow missed this email. The only test that GeoSpark does >>>>>> not cover for JTSplus is this one: >>>>>> https://github.com/jiayuasu/JTSplus/blob/master/src/test/java/jtsplustest/GeometryToStringTest.java >>>>>> >>>>>> If you can add this back to GeoSpark, I think you are good to go. >>>>>> >>>>>> Thanks, >>>>>> Jia >>>>>> >>>>>> On Thu, Jul 23, 2020 at 6:08 AM Netanel Malka >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> Have you had time to look at this? >>>>>>> >>>>>>> Best regards, >>>>>>> Netanel Malka. >>>>>>> >>>>>>> -- Forwarded message - >>>>>>> From: Netanel Malka >>>>>>> Date: Tue, 7 Jul 2020 at 11:06 >>>>>>> Subject: Re: Use JTS as a dependency instead of JTSPlus >>>>>>> To: Jia Yu >>>>>>> >>>>>>> >>>>>>> OK. >>>>>>> We saw that in Geometry the userData field changed from null to "", >>>>>>> is it crucial? because this is a change that I believe that JTS won't >>>>>>> accept. >>>>>>> >>>>>>> Also, does GeoSpark tests are covered JTSPlus changes? If all the >>>>>>> geospark tests are working, does it mean that we didn't break anything? >>>>>>> >>>>>>> >>>>>>> On Thu, 2
