Sorry yeah good question. It happens on the call to spark.table("pixels")
On Mon, Nov 1, 2021 at 12:36 PM Wenchen Fan <cloud0...@gmail.com> wrote:
> To confirm: Does the error happen during view creation, or when we read
> the view later?
>
> On Mon, Nov 1, 2021 at 11:28 PM Adam Binford <adam...@gmail.com> wrote:
>
>> I don't have a minimal reproduction right now but here's more relevant
>> code snippets.
>>
>> Stacktrace:
>> org.apache.spark.sql.AnalysisException: Undefined function:
>> 'ST_PolygonFromEnvelope'. This function is neither a registered temporary
>> function nor a permanent function registered in the database 'default'.;
>> line 2 pos 50
>> at
>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.failFunctionLookup(SessionCatalog.scala:1562)
>> at
>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1660)
>> at
>> org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1677)
>> at
>> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$27$$anonfun$applyOrElse$114.$anonfun$applyOrElse$116(Analyzer.scala:2150)
>> at
>> org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:60)
>> at
>> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$27$$anonfun$applyOrElse$114.applyOrElse(Analyzer.scala:2150)
>> at
>> org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$27$$anonfun$applyOrElse$114.applyOrElse(Analyzer.scala:2137)
>> at
>> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
>> at
>> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
>> at
>> org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
>>
>> Expression definition:
>> case class ST_PolygonFromEnvelope(inputExpressions: Seq[Expression])
>> extends Expression with CodegenFallback with UserDataGeneratator {
>> override def nullable: Boolean = false
>>
>> override def eval(input: InternalRow): Any = {
>> val minX = inputExpressions(0).eval(input) match {
>> case a: Double => a
>> case b: Decimal => b.toDouble
>> }
>>
>> val minY = inputExpressions(1).eval(input) match {
>> case a: Double => a
>> case b: Decimal => b.toDouble
>> }
>>
>> val maxX = inputExpressions(2).eval(input) match {
>> case a: Double => a
>> case b: Decimal => b.toDouble
>> }
>>
>> val maxY = inputExpressions(3).eval(input) match {
>> case a: Double => a
>> case b: Decimal => b.toDouble
>> }
>>
>> var coordinates = new Array[Coordinate](5)
>> coordinates(0) = new Coordinate(minX, minY)
>> coordinates(1) = new Coordinate(minX, maxY)
>> coordinates(2) = new Coordinate(maxX, maxY)
>> coordinates(3) = new Coordinate(maxX, minY)
>> coordinates(4) = coordinates(0)
>> val geometryFactory = new GeometryFactory()
>> val polygon = geometryFactory.createPolygon(coordinates)
>> new GenericArrayData(GeometrySerializer.serialize(polygon))
>> }
>>
>> override def dataType: DataType = GeometryUDT
>>
>> override def children: Seq[Expression] = inputExpressions
>> }
>>
>> Function registration:
>> Catalog.expressions.foreach(f => {
>> val functionIdentifier =
>> FunctionIdentifier(f.getClass.getSimpleName.dropRight(1))
>> val expressionInfo = new ExpressionInfo(
>> f.getClass.getCanonicalName,
>> functionIdentifier.database.orNull,
>> functionIdentifier.funcName)
>> sparkSession.sessionState.functionRegistry.registerFunction(
>> functionIdentifier,
>> expressionInfo,
>> f
>> )
>> })
>>
>> On Mon, Nov 1, 2021 at 10:43 AM Wenchen Fan <cloud0...@gmail.com> wrote:
>>
>>> Hi Adam,
>>>
>>> Thanks for reporting this issue! Do you have the full stacktrace or a
>>> code snippet to reproduce the issue at Spark side? It looks like a bug, but
>>> it's not obvious to me how this bug can happen.
>>>
>>> Thanks,
>>> Wenchen
>>>
>>> On Sat, Oct 30, 2021 at 1:08 AM Adam Binford <adam...@gmail.com> wrote:
>>>
>>>> Hi devs,
>>>>
>>>> I'm working on getting Apache Sedona upgraded to work with Spark 3.2,
>>>> and ran into a weird issue I wanted to get some feedback on. The PR and
>>>> current discussion can be found here:
>>>> https://github.com/apache/incubator-sedona/pull/557
>>>>
>>>> To try to sum up in a quick way, this library defines custom
>>>> expressions and registers the expressions using
>>>> sparkSession.sessionState.functionRegistry.registerFunction. One of
>>>> the unit tests is now failing because the function can't be found when a
>>>> temporary view using that function is created in pure SQL.
>>>>
>>>> Examples:
>>>> This fails with Undefined function: 'ST_PolygonFromEnvelope'. This
>>>> function is neither a registered temporary function nor a permanent
>>>> function registered in the database 'default'.:
>>>>
>>>> spark.sql(
>>>> """
>>>> |CREATE OR REPLACE TEMP VIEW pixels AS
>>>> |SELECT pixel, shape FROM pointtable
>>>> |LATERAL VIEW EXPLODE(ST_Pixelize(shape, 1000, 1000,
>>>> ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS pixel
>>>> """.stripMargin)
>>>>
>>>> // Test visualization partitioner
>>>> val zoomLevel = 2
>>>> val newDf = VizPartitioner(spark.table("pixels"), zoomLevel,
>>>> "pixel", new Envelope(0, 1000, 0, 1000))
>>>>
>>>>
>>>> But both of these work fine:
>>>>
>>>> val table = spark.sql(
>>>> """
>>>> |SELECT pixel, shape FROM pointtable
>>>> |LATERAL VIEW EXPLODE(ST_Pixelize(shape, 1000, 1000,
>>>> ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS pixel
>>>> """.stripMargin)
>>>>
>>>> // Test visualization partitioner
>>>> val zoomLevel = 2
>>>> val newDf = VizPartitioner(table, zoomLevel, "pixel", new
>>>> Envelope(0, 1000, 0, 1000))
>>>>
>>>> val table = spark.sql(
>>>> """
>>>> |SELECT pixel, shape FROM pointtable
>>>> |LATERAL VIEW EXPLODE(ST_Pixelize(shape, 1000, 1000,
>>>> ST_PolygonFromEnvelope(-126.790180,24.863836,-64.630926,50.000))) AS pixel
>>>> """.stripMargin)
>>>> table.createOrReplaceTempView("pixels")
>>>>
>>>> // Test visualization partitioner
>>>> val zoomLevel = 2
>>>> val newDf = VizPartitioner(spark.table("pixels"), zoomLevel,
>>>> "pixel", new Envelope(0, 1000, 0, 1000))
>>>>
>>>>
>>>> So the main question is, is this a feature or a bug?
>>>>
>>>> --
>>>> Adam Binford
>>>>
>>>
>>
>> --
>> Adam Binford
>>
>
--
Adam Binford
