Hi Adam, Thanks for reporting this issue! Do you have the full stacktrace or a code snippet to reproduce the issue at Spark side? It looks like a bug, but it's not obvious to me how this bug can happen.
Thanks, Wenchen On Sat, Oct 30, 2021 at 1:08 AM Adam Binford <adam...@gmail.com> wrote: > Hi devs, > > I'm working on getting Apache Sedona upgraded to work with Spark 3.2, and > ran into a weird issue I wanted to get some feedback on. The PR and current > discussion can be found here: > https://github.com/apache/incubator-sedona/pull/557 > > To try to sum up in a quick way, this library defines custom expressions > and registers the expressions using > sparkSession.sessionState.functionRegistry.registerFunction. One of the > unit tests is now failing because the function can't be found when a > temporary view using that function is created in pure SQL. > > Examples: > This fails with Undefined function: 'ST_PolygonFromEnvelope'. This > function is neither a registered temporary function nor a permanent > function registered in the database 'default'.: > > spark.sql( > """ > |CREATE OR REPLACE TEMP VIEW pixels AS > |SELECT pixel, shape FROM pointtable > |LATERAL VIEW EXPLODE(ST_Pixelize(shape, 1000, 1000, > ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS pixel > """.stripMargin) > > // Test visualization partitioner > val zoomLevel = 2 > val newDf = VizPartitioner(spark.table("pixels"), zoomLevel, "pixel", > new Envelope(0, 1000, 0, 1000)) > > > But both of these work fine: > > val table = spark.sql( > """ > |SELECT pixel, shape FROM pointtable > |LATERAL VIEW EXPLODE(ST_Pixelize(shape, 1000, 1000, > ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS pixel > """.stripMargin) > > // Test visualization partitioner > val zoomLevel = 2 > val newDf = VizPartitioner(table, zoomLevel, "pixel", new Envelope(0, > 1000, 0, 1000)) > > val table = spark.sql( > """ > |SELECT pixel, shape FROM pointtable > |LATERAL VIEW EXPLODE(ST_Pixelize(shape, 1000, 1000, > ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS pixel > """.stripMargin) > table.createOrReplaceTempView("pixels") > > // Test visualization partitioner > val zoomLevel = 2 > val newDf = VizPartitioner(spark.table("pixels"), zoomLevel, "pixel", > new Envelope(0, 1000, 0, 1000)) > > > So the main question is, is this a feature or a bug? > > -- > Adam Binford >
