Re: Support Spark 2.4 in Sedona 1.0

2020-11-12 Thread Felix Cheung 
You might be able to create shim and build in profile
Like
https://github.com/graphframes/graphframes/tree/branch-0.5/src/main


On Thu, Nov 12, 2020 at 3:23 PM Jia Yu  wrote:

> In order to support Spark 2.4, Sedona needs to use different logic for SQL
> aggregation functions. I am not sure if this could be achieved by using
> different profiles.
>
> Sedona for Spark 3.0:
> https://github.com/apache/incubator-sedona/blob/master/sql/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/AggregateFunctions.scala
> Sedona for Spark 2.4:
> https://github.com/apache/incubator-sedona/blob/spark-2.3-2.4/sql/src/main/scala/org/apache/spark/sql/geosparksql/expressions/AggregateFunctions.scala
>
> On Thu, Nov 12, 2020 at 9:42 AM Mo Sarwat  wrote:
>
>> Yes, I believe we still need to support Spark 2.4 since  many Sedona users
>> are still using it
>>
>> On Wed, Nov 11, 2020 at 11:21 PM Netanel Malka 
>> wrote:
>>
>> > What do you mean by compile target?
>> >
>> > ?
>> >
>> > Iv'e found Apache Zeppelin handle multiple Spark versions here using
>> > profile for each Spark version:
>> >
>> > https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L185
>> >
>> >
>> > Do you think this method is better?
>> >
>> >
>> >
>> > Netanel Malka,
>> > Big Data Consultant
>> > [Description: Description: Description: Description:
>> > cid:image001.jpg@01C85203.36A2AF30]
>> > 
>> > From: Felix Cheung 
>> > Sent: Thursday, November 12, 2020 04:05
>> > To: dev@sedona.apache.org
>> > Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi
>> > Zhang; d...@sedona.incubator.apache.org
>> > Subject: Re: Support Spark 2.4 in Sedona 1.0
>> >
>> > I am not sure it should be a branch? It is common to deal with this as a
>> > compile target, not as a separate branch. A separate branch might have
>> > difficulty to release?
>> >
>> > There are a few example in projects where they handle multiple Spark
>> > target version like this.
>> >
>> >
>> > On Wed, Nov 11, 2020 at 12:56 PM Jia Yu > > ji...@apache.org>> wrote:
>> > OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the
>> > compiler used in each branch,
>> >
>> > For Sedona on Spark 3.0, I will compile it using Scala 2.12
>> > For Sedona on Spark 2.4, I will compile it using Scala 2.11.
>> >
>> > For the Java code in both branches, I will compile them using Java 1.8
>> >
>> > Am I missing anything here?
>> >
>> >
>> > On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka > > > netan...@sela.co.il>> wrote:
>> >
>> > > Hi,
>> > >
>> > > I also think that we need to support 2.4.
>> > >
>> > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7
>> Sep
>> > > 12, 2020)
>> > >
>> > > I also asked about it on *us...@spark.apache.org> > us...@spark.apache.org> > > us...@spark.apache.org>>*
>> > >  :
>> > >
>> > >
>> > > *Sean Owen (answered the question): *
>> > >
>> > > "I don't think there's an official EOL for Spark 2.4.x but would
>> expect
>> > > another maintenance release in the first half of 2021 at least. I'd
>> also
>> > > guess it wouldn't be maintained by 2022."
>> > >
>> > >
>> > > ?BR,
>> > >
>> > >
>> > >
>> > > Netanel Malka,
>> > > Big Data Consultant
>> > > [image: Description: Description: Description: Description:
>> > > cid:image001.jpg@01C85203.36A2AF30]
>> > > --
>> > > *From:* Paweł Kociński > > pawel93kocin...@gmail.com>>
>> > > *Sent:* Wednesday, November 11, 2020 00:29
>> > > *To:* Jia Yu
>> > > *Cc:* dev@sedona.apache.org;
>> > d...@sedona.incubator.apache.org> >;
>> > Jinxuan Wu;
>> > > Mohamed Sarwat; Netanel Malka; Zongsi Zhang
>> > > *Subject:* Re: Support Spark 2.4 in Sedona 1.0
>> > >
>> > > Hi Jia,
>> > > I think we should support spark 2.4, a lot of users still use it. More
>> > > than that I think more users still have jobs written in spark 2.4 than
>> > > 3.0.  We will use an additional branch for that use case ? I mean
>> Spark
>> > 2.4
>> > > with scala 2.12 is important one.
>> > > Regards,
>> > > Paweł
>> > >
>> > > pon., 9 lis 2020 o 20:44 Jia Yu > > ji...@apache.org>> napisał(a):
>> > >
>> > >> Dear all,
>> > >>
>> > >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder
>> > whether
>> > >> we should support Spark 2.4.
>> > >>
>> > >> In order to support Spark 2.4, we need to do the following
>> > >>
>> > >> 1. Compile the source using Scala 2.11. Sedona master branch
>> currently
>> > is
>> > >> compiled by Scala 2.12 and Java 1.8
>> > >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the
>> > (1)
>> > >> UDF registration hook (2) the SQL aggregation function format
>> > >> 3. In the future releases of Sedona, use git cherry-pick to pick
>> > >> important features back to the Spark 2.4 branch. This is what I did
>> in
>> > >> GeoSpark to support Spark 2.1, 2.2, 2.3
>> > >>
>> > >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can

Re: Support Spark 2.4 in Sedona 1.0

2020-11-12 Thread Jia Yu 
In order to support Spark 2.4, Sedona needs to use different logic for SQL
aggregation functions. I am not sure if this could be achieved by using
different profiles.

Sedona for Spark 3.0:
https://github.com/apache/incubator-sedona/blob/master/sql/src/main/scala/org/apache/spark/sql/sedona_sql/expressions/AggregateFunctions.scala
Sedona for Spark 2.4:
https://github.com/apache/incubator-sedona/blob/spark-2.3-2.4/sql/src/main/scala/org/apache/spark/sql/geosparksql/expressions/AggregateFunctions.scala

On Thu, Nov 12, 2020 at 9:42 AM Mo Sarwat  wrote:

> Yes, I believe we still need to support Spark 2.4 since  many Sedona users
> are still using it
>
> On Wed, Nov 11, 2020 at 11:21 PM Netanel Malka 
> wrote:
>
> > What do you mean by compile target?
> >
> > ?
> >
> > Iv'e found Apache Zeppelin handle multiple Spark versions here using
> > profile for each Spark version:
> >
> > https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L185
> >
> >
> > Do you think this method is better?
> >
> >
> >
> > Netanel Malka,
> > Big Data Consultant
> > [Description: Description: Description: Description:
> > cid:image001.jpg@01C85203.36A2AF30]
> > 
> > From: Felix Cheung 
> > Sent: Thursday, November 12, 2020 04:05
> > To: dev@sedona.apache.org
> > Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi
> > Zhang; d...@sedona.incubator.apache.org
> > Subject: Re: Support Spark 2.4 in Sedona 1.0
> >
> > I am not sure it should be a branch? It is common to deal with this as a
> > compile target, not as a separate branch. A separate branch might have
> > difficulty to release?
> >
> > There are a few example in projects where they handle multiple Spark
> > target version like this.
> >
> >
> > On Wed, Nov 11, 2020 at 12:56 PM Jia Yu  > ji...@apache.org>> wrote:
> > OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the
> > compiler used in each branch,
> >
> > For Sedona on Spark 3.0, I will compile it using Scala 2.12
> > For Sedona on Spark 2.4, I will compile it using Scala 2.11.
> >
> > For the Java code in both branches, I will compile them using Java 1.8
> >
> > Am I missing anything here?
> >
> >
> > On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka   > netan...@sela.co.il>> wrote:
> >
> > > Hi,
> > >
> > > I also think that we need to support 2.4.
> > >
> > > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep
> > > 12, 2020)
> > >
> > > I also asked about it on *us...@spark.apache.org > us...@spark.apache.org>  > us...@spark.apache.org>>*
> > >  :
> > >
> > >
> > > *Sean Owen (answered the question): *
> > >
> > > "I don't think there's an official EOL for Spark 2.4.x but would expect
> > > another maintenance release in the first half of 2021 at least. I'd
> also
> > > guess it wouldn't be maintained by 2022."
> > >
> > >
> > > ?BR,
> > >
> > >
> > >
> > > Netanel Malka,
> > > Big Data Consultant
> > > [image: Description: Description: Description: Description:
> > > cid:image001.jpg@01C85203.36A2AF30]
> > > --
> > > *From:* Paweł Kociński  > pawel93kocin...@gmail.com>>
> > > *Sent:* Wednesday, November 11, 2020 00:29
> > > *To:* Jia Yu
> > > *Cc:* dev@sedona.apache.org;
> > d...@sedona.incubator.apache.org;
> > Jinxuan Wu;
> > > Mohamed Sarwat; Netanel Malka; Zongsi Zhang
> > > *Subject:* Re: Support Spark 2.4 in Sedona 1.0
> > >
> > > Hi Jia,
> > > I think we should support spark 2.4, a lot of users still use it. More
> > > than that I think more users still have jobs written in spark 2.4 than
> > > 3.0.  We will use an additional branch for that use case ? I mean Spark
> > 2.4
> > > with scala 2.12 is important one.
> > > Regards,
> > > Paweł
> > >
> > > pon., 9 lis 2020 o 20:44 Jia Yu  > ji...@apache.org>> napisał(a):
> > >
> > >> Dear all,
> > >>
> > >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder
> > whether
> > >> we should support Spark 2.4.
> > >>
> > >> In order to support Spark 2.4, we need to do the following
> > >>
> > >> 1. Compile the source using Scala 2.11. Sedona master branch currently
> > is
> > >> compiled by Scala 2.12 and Java 1.8
> > >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the
> > (1)
> > >> UDF registration hook (2) the SQL aggregation function format
> > >> 3. In the future releases of Sedona, use git cherry-pick to pick
> > >> important features back to the Spark 2.4 branch. This is what I did in
> > >> GeoSpark to support Spark 2.1, 2.2, 2.3
> > >>
> > >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave
> it
> > >> that way and just support Spark 3.0.
> > >>
> > >> Do you think we should support Spark 2.4 in the future release?
> > >>
> > >> Thanks,
> > >> Jia Yu
> > >>
> > >
> >
>

Re: First Sedona release

2020-11-12 Thread Jim Hughes 

Hi Mo,

I can definitely help.  The first step will be for Jia to push a PR for 
the JTS changes.  (Since they are his changes, I cannot do this on his 
behalf.)


From talking to the lead JTS developer, he wanted to see the previous 
PR (from months/a year+ ago) split up.  I think the initial PR should be 
used to discuss what changes are sensible for JTS and where we'll need 
to push some of the changes to Sedona.


Concretely, I noticed that the Sedona JTS fork changes the toString on 
Geometry to include printing out the userData.  I imagine that may cause 
trouble for downstream JTS users, so it'd be good to find an 
alternative.  One suggestion would to be add a static method in Sedona 
for printing a Geometry with its userData object.


Cheers,

Jim

On 11/12/20 12:32 PM, Mohamed Sarwat wrote:

Folks,

I totally agree with Jim on that. Jim, would you like to take the lead on that 
- I trust that you can bring this task to completion. Jia, would you please let 
us know how we can incorporate the changes into the JTS master branch?

Thanks,


On Nov 12, 2020, at 10:10 AM, Jim Hughes  wrote:

Hi all,

As a JTS committer, I have tried to request that the Sedona project discuss the 
desired changes to JTS previously.  I'd still encourage that.

JTS is an active project and I feel that maintaining a fork of JTS is 
unnecessary and inappropriate.

Cheers,

Jim


On 11/11/20 9:04 PM, Felix Cheung wrote:
Ah. You will need to publish it in order for the dependency chain to work
on Maven Central

However, since you are not the project owner there you might need to
publish that under a different artifact id.

In general, it would be best to avoid hard forking another project like
this.



On Wed, Nov 11, 2020 at 1:05 PM Jia Yu  wrote:

Hi Netanel,

That links to this git submodule:
https://github.com/jiayuasu/jts/blob/1.16.x/modules/core/pom.xml#L6

I can easily fix this by changing the version number here to 1.16.2
excluding "SNAPSHOT":
https://github.com/jiayuasu/jts/blob/1.16.x/modules/core/pom.xml#L6

Will this solve the problem?

On Wed, Nov 11, 2020 at 7:40 AM Netanel Malka 
wrote:


Hi Folks,

I tried to make a release (dry-run) following by
publishing-maven-artifacts
, and I
encountered an issue.

On sedona-core, we have jts-core as a dependency with the SNAPSHOT
version.
(link
<
https://github.com/apache/incubator-sedona/blob/2e60fc07b0eae78ccae3876d970e677fc9319c40/core/pom.xml#L37
)

As a prerequisite to the release process, we cannot have dependencies in a
SNAPSHOT version.


Do you have any clue about how to solve this?


On Mon, 9 Nov 2020 at 21:22, Netanel Malka  wrote:


OK. Thanks Felix.


Updates:

   *
   *   Opened a ticket for INFRA to Enable Nexus Access For Sedona<
https://issues.apache.org/jira/browse/INFRA-21085>
   *   Followed this<
https://infra.apache.org/publishing-maven-artifacts.html> guide to test
the maven release process
   *   I hope to create a PR soon for adjusting the build to deploy to

the

ASF Nexus repository
   *   The key that signs the artifacts were created and tested.

Do we want to create a candidate release for the current master branch?

Netanel Malka,
Big Data Consultant
[Description: Description: Description: Description:
cid:image001.jpg@01C85203.36A2AF30]

From: Felix Cheung 
Sent: Wednesday, November 4, 2020 19:57
To: dev@sedona.apache.org
Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi

Zhang

Subject: Re: First Sedona release

1) No you don’t need KEYS file in github only on the release share
https://dist.apache.org/repos/dist/dev/incubator/

2) as podling you add to
https://dist.apache.org/repos/dist/dev/incubator/
When you commit via svn you will be able to add a “directory” for Sedona

2a) for release, you basically do a svn rename to move from dev to

release

“path”

3) if you have java based artifacts, yes. You will publish to Nexus,
staging first and when release is signed off, you can click on the
interface to make it official, which then automatically sync to Maven
central.

Here is a script for example that does release signing and publication

to

Nexus (and staging before release)



https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh

On Wed, Nov 4, 2020 at 2:50 AM Netanel Malka 

netanel...@gmail.com>> wrote:
Hi,

I followed the release-signing
 doc and created a key

for

signing and hashing.

I have a few questions:

1. Should the KEYS file also be added to the project root directory

on

Github? ( I saw it in Apache Ant)
2. I saw in release-policy_upload-ci
 that we
need
to add a release candidate to

https://dist.apache.org/repos/dist/*dev*/

/. However, there does not seem to be a directory with Sedona as
the
TLP name. How may we be able to get a directory with that

Re: Support Spark 2.4 in Sedona 1.0

2020-11-12 Thread Mo Sarwat 
Yes, I believe we still need to support Spark 2.4 since  many Sedona users
are still using it

On Wed, Nov 11, 2020 at 11:21 PM Netanel Malka  wrote:

> What do you mean by compile target?
>
> ?
>
> Iv'e found Apache Zeppelin handle multiple Spark versions here using
> profile for each Spark version:
>
> https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L185
>
>
> Do you think this method is better?
>
>
>
> Netanel Malka,
> Big Data Consultant
> [Description: Description: Description: Description:
> cid:image001.jpg@01C85203.36A2AF30]
> 
> From: Felix Cheung 
> Sent: Thursday, November 12, 2020 04:05
> To: dev@sedona.apache.org
> Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi
> Zhang; d...@sedona.incubator.apache.org
> Subject: Re: Support Spark 2.4 in Sedona 1.0
>
> I am not sure it should be a branch? It is common to deal with this as a
> compile target, not as a separate branch. A separate branch might have
> difficulty to release?
>
> There are a few example in projects where they handle multiple Spark
> target version like this.
>
>
> On Wed, Nov 11, 2020 at 12:56 PM Jia Yu  ji...@apache.org>> wrote:
> OK. I agree. I am gonna create a branch for spark-2.3/2.4. Regarding the
> compiler used in each branch,
>
> For Sedona on Spark 3.0, I will compile it using Scala 2.12
> For Sedona on Spark 2.4, I will compile it using Scala 2.11.
>
> For the Java code in both branches, I will compile them using Java 1.8
>
> Am I missing anything here?
>
>
> On Wed, Nov 11, 2020 at 7:31 AM Netanel Malka  netan...@sela.co.il>> wrote:
>
> > Hi,
> >
> > I also think that we need to support 2.4.
> >
> > I saw that even Apache Spark still releases 2.4.x artifacts. (2.4.7 Sep
> > 12, 2020)
> >
> > I also asked about it on *us...@spark.apache.org us...@spark.apache.org>  us...@spark.apache.org>>*
> >  :
> >
> >
> > *Sean Owen (answered the question): *
> >
> > "I don't think there's an official EOL for Spark 2.4.x but would expect
> > another maintenance release in the first half of 2021 at least. I'd also
> > guess it wouldn't be maintained by 2022."
> >
> >
> > ?BR,
> >
> >
> >
> > Netanel Malka,
> > Big Data Consultant
> > [image: Description: Description: Description: Description:
> > cid:image001.jpg@01C85203.36A2AF30]
> > --
> > *From:* Paweł Kociński  pawel93kocin...@gmail.com>>
> > *Sent:* Wednesday, November 11, 2020 00:29
> > *To:* Jia Yu
> > *Cc:* dev@sedona.apache.org;
> d...@sedona.incubator.apache.org;
> Jinxuan Wu;
> > Mohamed Sarwat; Netanel Malka; Zongsi Zhang
> > *Subject:* Re: Support Spark 2.4 in Sedona 1.0
> >
> > Hi Jia,
> > I think we should support spark 2.4, a lot of users still use it. More
> > than that I think more users still have jobs written in spark 2.4 than
> > 3.0.  We will use an additional branch for that use case ? I mean Spark
> 2.4
> > with scala 2.12 is important one.
> > Regards,
> > Paweł
> >
> > pon., 9 lis 2020 o 20:44 Jia Yu  ji...@apache.org>> napisał(a):
> >
> >> Dear all,
> >>
> >> In Sedona 1.0, we definitely will support Spark 3.0. But I wonder
> whether
> >> we should support Spark 2.4.
> >>
> >> In order to support Spark 2.4, we need to do the following
> >>
> >> 1. Compile the source using Scala 2.11. Sedona master branch currently
> is
> >> compiled by Scala 2.12 and Java 1.8
> >> 2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the
> (1)
> >> UDF registration hook (2) the SQL aggregation function format
> >> 3. In the future releases of Sedona, use git cherry-pick to pick
> >> important features back to the Spark 2.4 branch. This is what I did in
> >> GeoSpark to support Spark 2.1, 2.2, 2.3
> >>
> >> GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it
> >> that way and just support Spark 3.0.
> >>
> >> Do you think we should support Spark 2.4 in the future release?
> >>
> >> Thanks,
> >> Jia Yu
> >>
> >
>

Re: First Sedona release

2020-11-12 Thread Mohamed Sarwat 
Folks, 

I totally agree with Jim on that. Jim, would you like to take the lead on that 
- I trust that you can bring this task to completion. Jia, would you please let 
us know how we can incorporate the changes into the JTS master branch?

Thanks,

> 
> On Nov 12, 2020, at 10:10 AM, Jim Hughes  wrote:
> 
> Hi all,
> 
> As a JTS committer, I have tried to request that the Sedona project discuss 
> the desired changes to JTS previously.  I'd still encourage that.
> 
> JTS is an active project and I feel that maintaining a fork of JTS is 
> unnecessary and inappropriate.
> 
> Cheers,
> 
> Jim
> 
>> On 11/11/20 9:04 PM, Felix Cheung wrote:
>> Ah. You will need to publish it in order for the dependency chain to work
>> on Maven Central
>> 
>> However, since you are not the project owner there you might need to
>> publish that under a different artifact id.
>> 
>> In general, it would be best to avoid hard forking another project like
>> this.
>> 
>> 
>>> On Wed, Nov 11, 2020 at 1:05 PM Jia Yu  wrote:
>>> 
>>> Hi Netanel,
>>> 
>>> That links to this git submodule:
>>> https://github.com/jiayuasu/jts/blob/1.16.x/modules/core/pom.xml#L6
>>> 
>>> I can easily fix this by changing the version number here to 1.16.2
>>> excluding "SNAPSHOT":
>>> https://github.com/jiayuasu/jts/blob/1.16.x/modules/core/pom.xml#L6
>>> 
>>> Will this solve the problem?
>>> 
>>> On Wed, Nov 11, 2020 at 7:40 AM Netanel Malka 
>>> wrote:
>>> 
 Hi Folks,
 
 I tried to make a release (dry-run) following by
 publishing-maven-artifacts
 , and I
 encountered an issue.
 
 On sedona-core, we have jts-core as a dependency with the SNAPSHOT
 version.
 (link
 <
 https://github.com/apache/incubator-sedona/blob/2e60fc07b0eae78ccae3876d970e677fc9319c40/core/pom.xml#L37
>>> 
 )
 
 As a prerequisite to the release process, we cannot have dependencies in a
 SNAPSHOT version.
 
 
 Do you have any clue about how to solve this?
 
 
 On Mon, 9 Nov 2020 at 21:22, Netanel Malka  wrote:
 
> OK. Thanks Felix.
> 
> 
> Updates:
> 
>   *
>   *   Opened a ticket for INFRA to Enable Nexus Access For Sedona<
> https://issues.apache.org/jira/browse/INFRA-21085>
>   *   Followed this<
> https://infra.apache.org/publishing-maven-artifacts.html> guide to test
> the maven release process
>   *   I hope to create a PR soon for adjusting the build to deploy to
 the
> ASF Nexus repository
>   *   The key that signs the artifacts were created and tested.
> 
> Do we want to create a candidate release for the current master branch?
> 
> Netanel Malka,
> Big Data Consultant
> [Description: Description: Description: Description:
> cid:image001.jpg@01C85203.36A2AF30]
> 
> From: Felix Cheung 
> Sent: Wednesday, November 4, 2020 19:57
> To: dev@sedona.apache.org
> Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi
 Zhang
> Subject: Re: First Sedona release
> 
> 1) No you don’t need KEYS file in github only on the release share
> https://dist.apache.org/repos/dist/dev/incubator/
> 
> 2) as podling you add to
> https://dist.apache.org/repos/dist/dev/incubator/
> When you commit via svn you will be able to add a “directory” for Sedona
> 
> 2a) for release, you basically do a svn rename to move from dev to
 release
> “path”
> 
> 3) if you have java based artifacts, yes. You will publish to Nexus,
> staging first and when release is signed off, you can click on the
> interface to make it official, which then automatically sync to Maven
> central.
> 
> Here is a script for example that does release signing and publication
 to
> Nexus (and staging before release)
> 
> 
 https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh
> 
> On Wed, Nov 4, 2020 at 2:50 AM Netanel Malka >>>  netanel...@gmail.com>> wrote:
> Hi,
> 
> I followed the release-signing
>  doc and created a key
 for
> signing and hashing.
> 
> I have a few questions:
> 
>1. Should the KEYS file also be added to the project root directory
 on
>Github? ( I saw it in Apache Ant)
>2. I saw in release-policy_upload-ci
> that we
> need
>to add a release candidate to
 https://dist.apache.org/repos/dist/*dev*/
> name>/. However, there does not seem to be a directory with Sedona as
> the
>TLP name. How may we be able to get a directory with that name? (Also
> for
>the *release*)
>3. Do we need to push the artifacts also to ASF Nexus Repository
 (beside
>

Re: First Sedona release

2020-11-12 Thread Jim Hughes 

Hi all,

As a JTS committer, I have tried to request that the Sedona project 
discuss the desired changes to JTS previously.  I'd still encourage that.


JTS is an active project and I feel that maintaining a fork of JTS is 
unnecessary and inappropriate.


Cheers,

Jim

On 11/11/20 9:04 PM, Felix Cheung wrote:

Ah. You will need to publish it in order for the dependency chain to work
on Maven Central

However, since you are not the project owner there you might need to
publish that under a different artifact id.

In general, it would be best to avoid hard forking another project like
this.


On Wed, Nov 11, 2020 at 1:05 PM Jia Yu  wrote:


Hi Netanel,

That links to this git submodule:
https://github.com/jiayuasu/jts/blob/1.16.x/modules/core/pom.xml#L6

I can easily fix this by changing the version number here to 1.16.2
excluding "SNAPSHOT":
https://github.com/jiayuasu/jts/blob/1.16.x/modules/core/pom.xml#L6

Will this solve the problem?

On Wed, Nov 11, 2020 at 7:40 AM Netanel Malka 
wrote:


Hi Folks,

I tried to make a release (dry-run) following by
publishing-maven-artifacts
, and I
encountered an issue.

On sedona-core, we have jts-core as a dependency with the SNAPSHOT
version.
(link
<
https://github.com/apache/incubator-sedona/blob/2e60fc07b0eae78ccae3876d970e677fc9319c40/core/pom.xml#L37



)

As a prerequisite to the release process, we cannot have dependencies in a
SNAPSHOT version.


Do you have any clue about how to solve this?


On Mon, 9 Nov 2020 at 21:22, Netanel Malka  wrote:


OK. Thanks Felix.


Updates:

   *
   *   Opened a ticket for INFRA to Enable Nexus Access For Sedona<
https://issues.apache.org/jira/browse/INFRA-21085>
   *   Followed this<
https://infra.apache.org/publishing-maven-artifacts.html> guide to test
the maven release process
   *   I hope to create a PR soon for adjusting the build to deploy to

the

ASF Nexus repository
   *   The key that signs the artifacts were created and tested.

Do we want to create a candidate release for the current master branch?

Netanel Malka,
Big Data Consultant
[Description: Description: Description: Description:
cid:image001.jpg@01C85203.36A2AF30]

From: Felix Cheung 
Sent: Wednesday, November 4, 2020 19:57
To: dev@sedona.apache.org
Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi

Zhang

Subject: Re: First Sedona release

1) No you don’t need KEYS file in github only on the release share
https://dist.apache.org/repos/dist/dev/incubator/

2) as podling you add to
https://dist.apache.org/repos/dist/dev/incubator/
When you commit via svn you will be able to add a “directory” for Sedona

2a) for release, you basically do a svn rename to move from dev to

release

“path”

3) if you have java based artifacts, yes. You will publish to Nexus,
staging first and when release is signed off, you can click on the
interface to make it official, which then automatically sync to Maven
central.

Here is a script for example that does release signing and publication

to

Nexus (and staging before release)



https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh


On Wed, Nov 4, 2020 at 2:50 AM Netanel Malka 

netanel...@gmail.com>> wrote:
Hi,

I followed the release-signing
 doc and created a key

for

signing and hashing.

I have a few questions:

1. Should the KEYS file also be added to the project root directory

on

Github? ( I saw it in Apache Ant)
2. I saw in release-policy_upload-ci
 that we
need
to add a release candidate to

https://dist.apache.org/repos/dist/*dev*/

/. However, there does not seem to be a directory with Sedona as
the
TLP name. How may we be able to get a directory with that name? (Also
for
the *release*)
3. Do we need to push the artifacts also to ASF Nexus Repository

(beside

Maven Central)?


Thanks.

On Mon, 2 Nov 2020 at 19:21, Netanel Malka 

netanel...@gmail.com>> wrote:


Thanks Felix.

I would be delighted to help.
I can start with the GPG.
  Can I test it on a some artifact, or I need to wait for the first

release?


On Mon, 2 Nov 2020 at 03:17, Felix Cheung 
> wrote:

Great progress!

To add,
A) I’d strongly recommend the WIP disclaimer - it would be much

easier

to

pass with in the first release
https://incubator.apache.org/policy/incubation.html#disclaimers

B) more info in signing, checksum
https://infra.apache.org/release-signing.html

C) signing key should be individual’s and (public key ) published and

also

listed in KEYS file - KEYS file  should be located next to the

staging

(and
later release) location, see above

D) “correct place” - this is in reference to ASF officIal staging

server

http://www.apache.org/legal/release-policy.html#stage
And can be “uploaded” by committing to svn