Hi all,
It may be worth discussing with the JTS directly what their schedule is
rather than guessing at it.
I am for finding a way for Sedona to work with JTS with the least
friction for the Sedona development team and the Sedona users. I feel
that copying or forking complex codebases will likely lead to bigger
issues downstream.
Also, is the only hang-up around the serialization of R-Trees? If so,
could you use reflection with JTS 1.17.0? That change may be pretty
quick to make...
Cheers,
Jim
On 12/9/20 10:35 PM, Jia Yu wrote:
Hi Felix, Jim and Netanel and other Sedona committers,
As you know, my JTS PR has been accepted to JTS 1.18-SNAPSHOT and we are
waiting for the official release of JTS 1.18 on Maven. However, I didn't
see a clear date when JTS 1.18 will be published. I guess this will take
one or two months to happen.
Currently, Sedona 1.0.0 release is blocked by this issue (Maven Central
does not allow SNAPSHOTS to be dependencies). Since we are so desperate
to
publish Sedona 1.0.0 as soon as possible, I proposed to copy the latest
JTS
source code into Sedona-core in our 1.0.0 release. In the future release
(say Sedona 1.0.1), we can drop JTS source code and use their Maven
release. JTS source code is dual-licensed under Eclipse Public License
2.0
and Eclipse Distribution License 1.0 (a BSD Style License). So it is safe
to keep it in Sedona.
What do you think? @Jim Hughes <jhug...@ccri.com> Is this a good idea?
Thanks,
Jia
On Fri, Dec 4, 2020 at 10:43 PM Jia Yu <jiayu198...@gmail.com> wrote:
Hi Netanel,
So for Sedona SQL 1.0.0 on Spark 2.4, we can do
"sedona-sql_2.11-2.4-1.0.0-incubator" , right?
Sedona 1.0 on Spark 2.4 and 3.0 will be compiled against Scala 2.11 and
2.12. I believe this can be done via different compilation target in
Maven.
I am currently looking at whether I can do conditional compilation using
Maven (similar to C++ #ifdef) because there is a change in Aggregator in
Spark 3.0. Otherwise I always need to maintain a separate branch for
Sedona
on Spark 2.4
It looks OK to me.
On Fri, Dec 4, 2020 at 1:12 AM Netanel Malka <netanel...@gmail.com>
wrote:
Hi,
I think that we can follow the Apache Spark convention as you can see
here
<
https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/3.0.1/>.
For example:
sedona-sql_2.11-2.4, where 2.11 -> scala version and 2.4 -> spark
version
What do you think?
On Fri, 4 Dec 2020 at 10:34, Jia Yu <jiayu198...@gmail.com> wrote:
Dear all,
The current status:
1. Move to JTS PR has been merged to the master branch. If JTS 1.18
gets
published in a few weeks, we will use the latest JTS. Otherwise, we
still
need to use my fork for this release. But Sedona API is now
finalized. From
the user perspective, use my fork or JTS official release should not
make
any difference.
2. Sedona doc update is in progress. I am half way there. You can
track
the progress here:
https://github.com/apache/incubator-sedona/pull/493
3. I will create a separate branch to test Spark 2.4 over this
weekend.
4. Pawel is working on his improvement on RDD-SQL Python adapter.
Question:
What is the most appropriate maven artifact name for Sedona on Spark
2.4? I used to put "sedona-sql_2.4". But it looks like "_2.4" is
usually
reserved for specifying the Scala version. How about
"sedona-sql-spark2"?
Should we also use "sedona-sql-spark3" for Spark 3.0?
Thanks,
Jia
On Tue, Nov 24, 2020 at 8:16 AM Jim Hughes <jhug...@ccri.com> wrote:
Hi all,
Felix, good to know that a WIP disclaimer is standard practice and
will
let things move forward!
Jia, I believe that page is explaining that a portion of the code in
various GeoTools modules has other licenses on it. As such, gt-main
is
mostly LGPL with some BSD code as well.
Cheers,
Jim
On 11/23/2020 9:50 PM, Jia Yu wrote:
Thank you, Felix. I will use the WIP disclaimer.
To answer Jim's question, GeoTools components use different
licenses:
https://docs.geotools.org/latest/userguide/welcome/license.html
GT-main uses BSD, so its binary can be included in Sedona's release.
Other components in GeoTools use LGPL, but Sedona only uses them for
CRS
transformation. I already set the dependency scope to "provided" in
Sedona's POM.xml. If a user wants to use CRS transformation in
Sedona, they
will have to add some GeoTools library by themselves.
On Mon, Nov 23, 2020 at 6:24 PM Felix Cheung <
felixche...@apache.org>
wrote:
On Mon, Nov 23, 2020 at 6:03 PM Felix Cheung <
felixche...@apache.org
wrote:
I’d strongly recommend the community to move towards the first
release
with the WIP disclaimer
https://incubator.apache.org/policy/incubation.html#work_in_progress_disclaimer
https://incubator.apache.org/policy/incubation.html#releases
As for the LGPL dependency specifically, a replacement will be
needed?
To clarify, ok to note in the WIP disclaimer- so it can be released
with
this.
On Mon, Nov 23, 2020 at 11:15 AM Jim Hughes <jhug...@ccri.com>
wrote:
Hi all,
Has the fact that one of the dependencies is LGPL (GeoTools) been
discussed / addressed? (See
https://www.apache.org/legal/resolved.html#category-x)
I'm asking since I don't know if the ASF has any recommended work
arounds for shipping code with licenses that it does not approve
of.
Cheers,
Jim
On 11/23/20 1:41 PM, Felix Cheung wrote:
I can help review around Dev 13 to give a first pass. It should
give
you an
easier path to IPMC vote.
On Sun, Nov 22, 2020 at 10:50 PM Jia Yu <jiayu198...@gmail.com>
wrote:
Hi Pawel and everyone,
Let's do this in the first Sedona release. But can you please
first
fix the
Python API for our Move-to-JTS PR, and then work on this one?
If
this
Python RDD-DF Adapter PR might slow down our progress of
releasing
Sedona
before Christmas, we can postpone it to Sedona 1.0.1 or 1.1.0.
@everyone
Our top priority is to draw the first Sedona release ASAP.
Users
have
been
waiting for almost six months. Let's push hard to publish the
first
Sedona
release to Maven Central and PyPI before Christmas. In order to
make
it
happen,
Finalize coding and documentation before Dec 6:
1. I believe the Move-to-JTS PR will be done in around one
week.
2. Then we can accept Pawel' Python RDD-DF Adapter PR, if
necessary
3. I will work on Sedona documentation.
4. @Netanel will work on Sedona support of Spark 2.4 and Scala
2.11.
I
will
first create a branch for it to illustrate some necessary
changes in
Sedona
SQL for Spark 2.4.
Final walk-through before Dec 13
1. Netanel can test the release management for Sedona.
2. Other committers can go through the docs, release notes
Community voting before Dec 20
1. Sedona community voting: before Dec 16
2. Apache Incubator voting: before Dec 20
Push to Maven Central and PyPi before Dec 24
Please feel free to comment if you have any suggestions!
Jia
On Sun, Nov 22, 2020 at 9:51 AM Paweł Kociński <
pawel93kocin...@gmail.com>
wrote:
Hi,
I saw some users reported need to improve Python RDD API in
two
scenarios:
- converting spatial flat join result to df
- saving spatial flat join result directly to external storage
Currently SerDe between jvm and Python causes additional time
needed
to
compute the result. I have a local branch with tests where
this
functionality is available (need 3-4 days to make it 100%
ready), in
two
above scenarios there will be almost no difference between
Python
and
Scala
or Java API. Should I create PR to include this feature within
the
first
Sedona release ?
Regards,
Paweł
pon., 16 lis 2020 o 08:29 Jia Yu <jiayu198...@gmail.com>
napisał(a):
Dear all,
Thanks for all your suggestions.
1. To completely solve the long-overdue JTS issue, I made a
Sedona
PR
and
two JTS PRs. @Jim Hughes <jhug...@ccri.com> , @Paweł
Kociński
<pawel93kocin...@gmail.com> , I, and probably Martin from
JTS
will
take
care of these PRs in the coming days.
(1) Sedona PR:
https://github.com/apache/incubator-sedona/pull/488
(2) JTS PR: https://github.com/locationtech/jts/pull/633
https://github.com/locationtech/jts/pull/634
2. To move forward with the first release, I have deleted the
"SNAPSHOT"
in my JTS 1.16 fork.
Most likely, we have to move forward with my JTS 1.16 fork in
the
first
Sedona release because of the conflict among JTStoGeoJSON,
GeoTools,
and
JTS 1.17.
So @Netanel Malka <netanel...@gmail.com> could you please
do
another
dry-run on the Sedona first release on this Sedona branch:
sedona-1.0-doc:
https://github.com/apache/incubator-sedona/tree/sedona-1.0-doc
Thanks,
Jia
On Thu, Nov 12, 2020 at 11:36 AM Jim Hughes <
jhug...@ccri.com>
wrote:
Hi Mo,
I can definitely help. The first step will be for Jia to
push a
PR
for
the JTS changes. (Since they are his changes, I cannot do
this on
his
behalf.)
From talking to the lead JTS developer, he wanted to see
the
previous
PR (from months/a year+ ago) split up. I think the initial
PR
should
be
used to discuss what changes are sensible for JTS and where
we'll
need
to push some of the changes to Sedona.
Concretely, I noticed that the Sedona JTS fork changes the
toString
on
Geometry to include printing out the userData. I imagine
that may
cause
trouble for downstream JTS users, so it'd be good to find an
alternative. One suggestion would to be add a static method
in
Sedona
for printing a Geometry with its userData object.
Cheers,
Jim
On 11/12/20 12:32 PM, Mohamed Sarwat wrote:
Folks,
I totally agree with Jim on that. Jim, would you like to
take the
lead
on that - I trust that you can bring this task to
completion.
Jia,
would
you please let us know how we can incorporate the changes
into the
JTS
master branch?
Thanks,
On Nov 12, 2020, at 10:10 AM, Jim Hughes <
jhug...@ccri.com>
wrote:
Hi all,
As a JTS committer, I have tried to request that the
Sedona
project
discuss the desired changes to JTS previously. I'd still
encourage
that.
JTS is an active project and I feel that maintaining a
fork
of
JTS
is
unnecessary and inappropriate.
Cheers,
Jim
On 11/11/20 9:04 PM, Felix Cheung wrote:
Ah. You will need to publish it in order for the
dependency
chain
to
work
on Maven Central
However, since you are not the project owner there you
might
need
to
publish that under a different artifact id.
In general, it would be best to avoid hard forking
another
project
like
this.
On Wed, Nov 11, 2020 at 1:05 PM Jia Yu <
jiayu198...@gmail.com
wrote:
Hi Netanel,
That links to this git submodule:
https://github.com/jiayuasu/jts/blob/1.16.x/modules/core/pom.xml#L6
I can easily fix this by changing the version number
here
to
1.16.2
excluding "SNAPSHOT":
https://github.com/jiayuasu/jts/blob/1.16.x/modules/core/pom.xml#L6
Will this solve the problem?
On Wed, Nov 11, 2020 at 7:40 AM Netanel Malka <
netanel...@gmail.com
wrote:
Hi Folks,
I tried to make a release (dry-run) following by
publishing-maven-artifacts
<
https://infra.apache.org/publishing-maven-artifacts.html>,
and
I
encountered an issue.
On sedona-core, we have jts-core as a dependency with
the
SNAPSHOT
version.
(link
<
https://github.com/apache/incubator-sedona/blob/2e60fc07b0eae78ccae3876d970e677fc9319c40/core/pom.xml#L37
)
As a prerequisite to the release process, we cannot
have
dependencies in a
SNAPSHOT version.
Do you have any clue about how to solve this?
On Mon, 9 Nov 2020 at 21:22, Netanel Malka <
netan...@sela.co.il>
wrote:
OK. Thanks Felix.
Updates:
*
* Opened a ticket for INFRA to Enable Nexus
Access For
Sedona<
https://issues.apache.org/jira/browse/INFRA-21085>
* Followed this<
https://infra.apache.org/publishing-maven-artifacts.html>
guide
to test
the maven release process
* I hope to create a PR soon for adjusting the
build
to
deploy to
the
ASF Nexus repository
* The key that signs the artifacts were
created
and
tested.
Do we want to create a candidate release for the
current
master
branch?
Netanel Malka,
Big Data Consultant
[Description: Description: Description: Description:
cid:image001.jpg@01C85203.36A2AF30]
________________________________
From: Felix Cheung <felixche...@apache.org>
Sent: Wednesday, November 4, 2020 19:57
To: dev@sedona.apache.org
Cc: Jinxuan Wu; Mohamed Sarwat; Netanel Malka; Paweł
Kociński;
Zongsi
Zhang
Subject: Re: First Sedona release
1) No you don’t need KEYS file in github only on the
release
share
https://dist.apache.org/repos/dist/dev/incubator/
2) as podling you add to
https://dist.apache.org/repos/dist/dev/incubator/
When you commit via svn you will be able to add a
“directory”
for
Sedona
2a) for release, you basically do a svn rename to move
from
dev
to
release
“path”
3) if you have java based artifacts, yes. You will
publish
to
Nexus,
staging first and when release is signed off, you can
click
on
the
interface to make it official, which then
automatically
sync
to
Maven
central.
Here is a script for example that does release signing
and
publication
to
Nexus (and staging before release)
https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh
On Wed, Nov 4, 2020 at 2:50 AM Netanel Malka <
netanel...@gmail.com
<mailto:
netanel...@gmail.com>> wrote:
Hi,
I followed the release-signing
<https://infra.apache.org/release-signing.html> doc
and
created
a key
for
signing and hashing.
I have a few questions:
1. Should the KEYS file also be added to the
project
root
directory
on
Github? ( I saw it in Apache Ant)
2. I saw in release-policy_upload-ci
<
http://www.apache.org/legal/release-policy.html#upload-ci>
that we
need
to add a release candidate to
https://dist.apache.org/repos/dist/*dev*/
<TLP
name>/. However, there does not seem to be a
directory
with
Sedona as
the
TLP name. How may we be able to get a directory
with
that
name? (Also
for
the *release*)
3. Do we need to push the artifacts also to ASF
Nexus
Repository
(beside
Maven Central)?
Thanks.
On Mon, 2 Nov 2020 at 19:21, Netanel Malka <
netanel...@gmail.com
<mailto:
netanel...@gmail.com>> wrote:
Thanks Felix.
I would be delighted to help.
I can start with the GPG.
Can I test it on a some artifact, or I need to
wait for
the
first
release?
On Mon, 2 Nov 2020 at 03:17, Felix Cheung <
felixche...@apache.org
<mailto:felixche...@apache.org>> wrote:
Great progress!
To add,
A) I’d strongly recommend the WIP disclaimer - it
would be
much
easier
to
pass with in the first release
https://incubator.apache.org/policy/incubation.html#disclaimers
B) more info in signing, checksum
https://infra.apache.org/release-signing.html
C) signing key should be individual’s and (public
key
)
published and
also
listed in KEYS file - KEYS file should be located
next to
the
staging
(and
later release) location, see above
D) “correct place” - this is in reference to ASF
officIal
staging
server
http://www.apache.org/legal/release-policy.html#stage
And can be “uploaded” by committing to svn
http://www.apache.org/legal/release-policy.html#upload-ci
E) python / PyPI -
https://incubator.apache.org/guides/distribution.html#pypi
On Sun, Nov 1, 2020 at 2:17 PM Jia Yu <
ji...@apache.org
<mailto:
ji...@apache.org>> wrote:
Hi Netanel, Pawel and other committers,
While Pawel is working on Python code of Sedona
1.0,
let's
focus on
other
parts required by the release. Netanel, can you
help
me
with
all
the
ASF
incubator requirement items that are not DONE?
*Here is a checklist for our first Sedona release*
*ASF incubator requirement
(
https://incubator.apache.org/guides/releasemanagement.html
<
https://incubator.apache.org/guides/releasemanagement.html
,
we
probably
should read ASF release requirement as well):*
1 .Include the word incubating in the release file
name:
DONE.
Please
see
the POM.xml in all directories.
2. Include an ASF LICENSE and NOTICE file: DONE.
Please
see
the
GitHub
repo.
3. Have valid checksums or signatures: I believe
signature
should
be
done
by the GPG key. Not sure about the checksum. I am
also
not
sure
about
the
GPG key requirement of ASF. I use GPG key to sign
releases
of
GeoSpark
in
the past.
4. Be placed in the correct place on the ASF’s
infrastructure:
we
should
place our releases in two places: Maven, and PyPi.
Not
sure
how to
relate
them to ASF.
5. Have a KEYS file to validate the release: this
should
be
the
public
key
of our GPG key?
*Sedona requirement*
1. Python path name, file headers, and jars
2. Project website docs: documentation should use
the
name,
Sedona, in
all
tutorials. We should also include the situation of
GeoTools
dependencies.
Thanks,
Jia
On Wed, Oct 14, 2020 at 10:08 PM Jia Yu <
ji...@apache.org
<mailto:
ji...@apache.org>> wrote:
Hi folks,
We will be working on the first Sedona. Please see
the
JIRA
ticket
here:
https://issues.apache.org/jira/projects/SEDONA/issues/SEDONA-3?filter=allopenissues
Do you think there are any outstanding issues to
be
fixed
as
well?
Thanks,
Jia
--
Best regards,
Netanel Malka.
--
Best regards,
Netanel Malka.
--
Best regards,
Netanel Malka.
--
Best regards,
Netanel Malka.
