Thank you for the follow up. Testing out those specific libraries, but not sure where to source the Geometry type from now, since I’m not using JTS anymore.
import org.locationtech.jts.geom.Geometry command-4367909143506860:37: error: not found: type Geometry var pointRDD = new SpatialRDD[Geometry]() ^ Thanks, Jerrod From: Jia Yu <[email protected]> Sent: Tuesday, February 23, 2021 7:02 PM To: Jerrod Wagnon <[email protected]>; [email protected] Subject: Re: Attribute Columns Question CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. BTW, in fact, you just need to use sedona-python-adapter, sedona-viz, geotools-wrapper, sernetcdf. JTS / jts2geojson jars are not needed. Then the JTS / jts2geojson conflict issue should be gone. On Tue, Feb 23, 2021 at 4:52 PM Jia Yu <[email protected]<mailto:[email protected]>> wrote: Jerrod, Just updated the doc. We were struggling to write docs for all these packaging issues in Sedona 1.0.0. The latest doc of Scala and Java dependency should be super clear: You need to open this link in an incognito window which automatically clears the old browser cache. http://sedona.apache.org/download/GeoSpark-All-Modules-Maven-Central-Coordinates/#spark-30-scala-212<https://urldefense.proofpoint.com/v2/url?u=http-3A__sedona.apache.org_download_GeoSpark-2DAll-2DModules-2DMaven-2DCentral-2DCoordinates_-23spark-2D30-2Dscala-2D212&d=DwMFaQ&c=MsXLK6sQQBUgeD0JbTyYgA&r=9azZCQem_NPy1XE-fJ5d4mblFkmnSjMQGyiXFG3JznU&m=yywm82BF4_20l0Wnhj2fn25WixwguBJs4N63R46vOek&s=G5zHcJlsAriSFXIbGzDLHsuLfrD9mNuwY7rB0snDRo4&e=> On Tue, Feb 23, 2021 at 2:01 PM Jia Yu <[email protected]<mailto:[email protected]>> wrote: Hi Jerrod, If you look into that code example, actually there is one more line before it: import scala.collection.JavaConversions._ Just add this line, you will be fine. Sedona 1.0.0 has to use JTS 1.18. You may have other errors later on. jts2geojson library, internally pacakge JTS 1.16. You will need to exclude it probably: https://github.com/bjornharrtell/jts2geojson/blob/master/pom.xml#L58<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bjornharrtell_jts2geojson_blob_master_pom.xml-23L58&d=DwMFaQ&c=MsXLK6sQQBUgeD0JbTyYgA&r=9azZCQem_NPy1XE-fJ5d4mblFkmnSjMQGyiXFG3JznU&m=yywm82BF4_20l0Wnhj2fn25WixwguBJs4N63R46vOek&s=l8ibdWi9EcMR83IYx1gy0Og14K760xp7gL_jroIk85c&e=> We will update our docs shortly On Tue, Feb 23, 2021 at 1:35 PM Jerrod Wagnon <[email protected]<mailto:[email protected]>> wrote: Thanks. There was a conflict with that method and the rdd.fieldNames (java list vs a sequence) when I tried to pass those in from the rdd. command-4367909143506860:54: error: type mismatch; found : java.util.List[String] required: Seq[String] var joinResultTest = Adapter.toDf(resultPairRDD, polygonRDD.fieldNames, pointRDD.fieldNames, spark) However, I was able to create the sequences manually and it seems to be working fine now. I had tried this previously, but had the geometry columns included in the sequence, which isn’t needed. var joinResultDf = Adapter.toDf(resultPairRDD, Seq("LocationCode"), Seq("UniqueID","EquipmentID","MKT_AREA"), spark) Also, in Databricks, I was having issues with the latest library for locationtech. 1.18 was having a conflict with wololo, so I ended up just loading 1.16 instead. That might not be a good solution, but it was the only way I could get all the dependencies to load properly on cluster restart. Just wanted to mention that in case anyone else runs into issues with Databricks Spark 3.0.1 and Scala 2.12. Everything seems to be working for my use cases with these libraries. [cid:[email protected]] Thanks again. Really appreciate the help and your quick response! Jerrod From: Jia Yu <[email protected]<mailto:[email protected]>> Sent: Tuesday, February 23, 2021 1:41 PM To: [email protected]<mailto:[email protected]>; Jerrod Wagnon <[email protected]<mailto:[email protected]>> Subject: Re: Attribute Columns Question CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Jerrod, Can you try to use this method: https://github.com/apache/incubator-sedona/blob/master/sql/src/test/scala/org/apache/sedona/sql/adapterTestScala.scala#L138<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dsedona_blob_master_sql_src_test_scala_org_apache_sedona_sql_adapterTestScala.scala-23L138&d=DwMFaQ&c=MsXLK6sQQBUgeD0JbTyYgA&r=9azZCQem_NPy1XE-fJ5d4mblFkmnSjMQGyiXFG3JznU&m=vZEjTroVnQm5j1w7ocpG9We1c_0EKseB4KvDXL9Noek&s=yHlmPPdrz1gjnP-gAHrZPweDJ_CSudJO0S3qRF3er6w&e=> Basically, you need to use rdd.fieldnames as input parameters. I think our doc missed this part. Our Adapter implementation for join query result is here: https://github.com/apache/incubator-sedona/blob/master/sql/src/main/scala/org/apache/sedona/sql/utils/Adapter.scala#L130<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dsedona_blob_master_sql_src_main_scala_org_apache_sedona_sql_utils_Adapter.scala-23L130&d=DwMFaQ&c=MsXLK6sQQBUgeD0JbTyYgA&r=9azZCQem_NPy1XE-fJ5d4mblFkmnSjMQGyiXFG3JznU&m=vZEjTroVnQm5j1w7ocpG9We1c_0EKseB4KvDXL9Noek&s=o7FT3PmYqmPFJrSKmNt4doxdcub3uUq4oaOerIv8xd0&e=> Thanks, Jia On Tue, Feb 23, 2021 at 9:13 AM Jerrod Wagnon <[email protected]<mailto:[email protected]>> wrote: I’m sure this is something simple I’m missing. Caveat, I’m not a developer, but can manage. Is there something different that needs to be done in Sedona vs the previous Snapshot version for Spark 3.0 to get additional columns to carry through in the JoinQuery.SpatialJoinQueryFlat results? Previously, I just passed the columns in with the Adapter.toSpatialRDD and they carried through. Now, I'm just just getting my two Geometry columns when converting back to a dataframe. I've tried passing the left and right field names into Adapter.toDf, but that results in an error when displaying the resulting dataframe. I’m using Scala in Databricks. I’ve read the online documentation, but can’t seem to find examples that help in this scenario. Sedona: Snapshot: Thanks, Jerrod This email contains confidential material for the sole use of the intended recipient(s). Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. This email contains confidential material for the sole use of the intended recipient(s). Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. This email contains confidential material for the sole use of the intended recipient(s). Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
