jnh5y commented on pull request #31461: URL: https://github.com/apache/spark/pull/31461#issuecomment-773355284
Hi all, LocationTech GeoMesa is a consumer of the previously package private UDT/UDF APIs in order implement spatial data types (like Point, LineString, Polygon) and associated functions (intersects, intersection, union, convexhull, etc.). I can appreciate the nature of the API developed here, and it has made good sense for it to private while forming. With Java 8, as a downstream consumer of the (private) API, we can do things like this: https://github.com/locationtech/geomesa/tree/main/geomesa-spark/geomesa-spark-sql/src/main/scala/org/apache/spark/sql and it works. Folks like @metasim and myself have watched the previous UDT PRs to see if those improvements would make it. With Java 11 and later caring about split packages, this issue is more important for these integrations. If something isn't done to address it, projects which are using the private API to implement UDT/UDFs are sunk on Java 11:(. This last time that the change came up, Simeon or Sean noted that just opening up these classes may allow us to move integration code out of the org.apache.spark packages. In terms of changes, if the community cannot settle on the proposed improvements in a timely fashion, I'd say that this is a great compromise. All that said, as a developer on a downstream project, I know that the integration UDTs/UDFs and the associated Catalyst optimizations will change routinely as Spark releases minor and major versions. (We have used reflection to paper over changes that have come up in the optimizer rules over time.) I think marking this as a DeveloperApi reflects the idea that things can change. I'd be quite happy to see this PR and other UDT improvements land in either order. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
