Support Spark 2.4 in Sedona 1.0

[Jia Yu]

Dear all,

In Sedona 1.0, we definitely will support Spark 3.0. But I wonder whether
we should support Spark 2.4.

In order to support Spark 2.4, we need to do the following

1. Compile the source using Scala 2.11. Sedona master branch currently is
compiled by Scala 2.12 and Java 1.8
2. For the Scala code of Sedona-SQL and Viz-SQL, I need to change the (1)
UDF registration hook (2) the SQL aggregation function format
3. In the future releases of Sedona, use git cherry-pick to pick important
features back to the Spark 2.4 branch. This is what I did in GeoSpark to
support Spark 2.1, 2.2, 2.3

GeoSpark 1.2.0 - 1.3.1 support Spark 2.4 already. We can simply leave it
that way and just support Spark 3.0.

Do you think we should support Spark 2.4 in the future release?

Thanks,
Jia Yu

Re: First Sedona release

[Netanel Malka]

OK. Thanks Felix.


Updates:

  *
  *   ​Opened a ticket for INFRA to ​Enable Nexus Access For 
Sedona
  *   Followed this 
guide to test the maven release process
  *   I hope to create a PR soon for adjusting the build to deploy to the ASF 
Nexus repository
  *   The key that signs the artifacts were created and tested.

Do we want to create a candidate release for the current master branch?
​
Netanel Malka,
Big Data Consultant
[Description: Description: Description: Description: 
cid:image001.jpg@01C85203.36A2AF30]

From: Felix Cheung 
Sent: Wednesday, November 4, 2020 19:57
To: dev@sedona.apache.org
Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł Kociński; Zongsi Zhang
Subject: Re: First Sedona release

1) No you don’t need KEYS file in github only on the release share
https://dist.apache.org/repos/dist/dev/incubator/

2) as podling you add to
https://dist.apache.org/repos/dist/dev/incubator/
When you commit via svn you will be able to add a “directory” for Sedona

2a) for release, you basically do a svn rename to move from dev to release 
“path”

3) if you have java based artifacts, yes. You will publish to Nexus, staging 
first and when release is signed off, you can click on the interface to make it 
official, which then automatically sync to Maven central.

Here is a script for example that does release signing and publication to Nexus 
(and staging before release)
https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh


On Wed, Nov 4, 2020 at 2:50 AM Netanel Malka 
mailto:netanel...@gmail.com>> wrote:
Hi,

I followed the release-signing
 doc and created a key for
signing and hashing.

I have a few questions:

   1. Should the KEYS file also be added to the project root directory on
   Github? ( I saw it in Apache Ant)
   2. I saw in release-policy_upload-ci
    that we need
   to add a release candidate to https://dist.apache.org/repos/dist/*dev*//. However, there does not seem to be a directory with Sedona as the
   TLP name. How may we be able to get a directory with that name? (Also for
   the *release*)
   3. Do we need to push the artifacts also to ASF Nexus Repository (beside
   Maven Central)?


Thanks.

On Mon, 2 Nov 2020 at 19:21, Netanel Malka 
mailto:netanel...@gmail.com>> wrote:

> Thanks Felix.
>
> I would be delighted to help.
> I can start with the GPG.
>  Can I test it on a some artifact, or I need to wait for the first release?
>
>
> On Mon, 2 Nov 2020 at 03:17, Felix Cheung 
> mailto:felixche...@apache.org>> wrote:
>
>> Great progress!
>>
>> To add,
>> A) I’d strongly recommend the WIP disclaimer - it would be much easier to
>> pass with in the first release
>> https://incubator.apache.org/policy/incubation.html#disclaimers
>>
>> B) more info in signing, checksum
>> https://infra.apache.org/release-signing.html
>>
>> C) signing key should be individual’s and (public key ) published and also
>> listed in KEYS file - KEYS file  should be located next to the staging
>> (and
>> later release) location, see above
>>
>> D) “correct place” - this is in reference to ASF officIal staging server
>> http://www.apache.org/legal/release-policy.html#stage
>> And can be “uploaded” by committing to svn
>> http://www.apache.org/legal/release-policy.html#upload-ci
>>
>> E) python / PyPI -
>> https://incubator.apache.org/guides/distribution.html#pypi
>>
>>
>>
>> On Sun, Nov 1, 2020 at 2:17 PM Jia Yu 
>> mailto:ji...@apache.org>> wrote:
>>
>> > Hi Netanel, Pawel and other committers,
>> >
>> > While Pawel is working on Python code of Sedona 1.0, let's focus on
>> other
>> > parts required by the release. Netanel, can you help me with all the ASF
>> > incubator requirement items that are not DONE?
>> >
>> > *Here is a checklist for our first Sedona release*
>> >
>> > *ASF incubator requirement
>> > (https://incubator.apache.org/guides/releasemanagement.html
>> > , we
>> probably
>> > should read ASF release requirement as well):*
>> >
>> > 1 .Include the word incubating in the release file name: DONE. Please
>> see
>> > the POM.xml in all directories.
>> >
>> > 2. Include an ASF LICENSE and NOTICE file: DONE. Please see the GitHub
>> > repo.
>> >
>> > 3. Have valid checksums or signatures: I believe signature should be
>> done
>> > by the GPG key. Not sure about the checksum. I am also not sure about
>> the
>> > GPG key requirement of ASF. I use GPG key to sign releases of GeoSpark
>> in
>> > the past.
>> >
>> > 4. Be placed in the correct place on the ASF’s infrastructure: we should
>> > place our releases in two places: Maven, and PyPi. Not sure how to
>> relate
>> > them to ASF.
>> >
>> > 5. Have a KEYS file to validate the release: this should be the public
>> key