Ok. That's why I didn't have this issue before. I am always using the PySpark kernel on EMR with Jupyter notebook... So in other words, does this only happen when people use the Scala spark kernel on EMR?
On Tue, Mar 28, 2023 at 3:54 AM Kristin Cowalcijk <[email protected]> wrote: > > Hi. > > I'm also aware of this issue. The ST_ functions provided by Apache Sedona > work fine when using spark.sql(...) or expr(...), however, the classes > provided by Apache Sedona cannot be imported when using the Spark kernel. > > This does not seem to be an issue of Apache Sedona, but an issue of AWS > EMR. I've also encountered other issues when using the Spark kernel on EMR > notebook (such as outputting to stderr breaks the entire kernel), so I > simply switched to using the PySpark kernel :-( > > Regards, > Kristin > > On Tue, Mar 28, 2023 at 6:07 AM PERSIE WANG <[email protected]> wrote: > > > To whom it may concern, > > > > I'm following your tutorial to run the spark on EMR notebook, it seems like > > there exists a dependency issue between Sedona and Geotools. See details as > > following, > > > > In order to use ST_transform function, I need both Sedona and > > Geotools-wrapper (for gt-referencing), so I follow the the tutorial as > > following: > > > > https://sedona.apache.org/latest-snapshot/setup/emr/#add-software-configuration > > to setup cluster. The spark application can be initiated, but when i import > > the following dependencies, it does not work. "*<console>:45: error: object > > sedona is not a member of package org"* > > > > *"* > > > > > > > > > > > > > > > > > > > > *import org.apache.spark.SparkContextimport > > org.apache.spark.sql.SparkSessionimport > > org.apache.spark.serializer.KryoSerializerimport > > org.apache.sedona.core.formatMapper.GeoJsonReaderimport > > org.apache.sedona.sql.utils.{Adapter, SedonaSQLRegistrator}import > > org.apache.sedona.core.serde.SedonaKryoRegistratorimport > > java.net.URLDecoderimport > > org.opengis.referencing.NoSuchAuthorityCodeExceptionimport > > org.locationtech.jts.geomimport org.wololo.geojson.GeoJSON* > > *"* > > > > I also tried use SparkMagic in my cells to use > > > > https://mvnrepository.com/artifact/org.apache.sedona/sedona-spark-shaded-3.0_2.12/1.4.0 > > and > > > > https://mvnrepository.com/artifact/org.datasyslab/geotools-wrapper/1.4.0-28.2 > > , > > but got the same error. But if i use sedona-shaded only, all imports works > > except *org.opengis.referencing.NoSuchAuthorityCodeException.* > > > > > > Please help take a look, and let me know if there is any tracking issue > > related. > > > > Thanks > > Percy > >
