viirya commented on pull request #31461: URL: https://github.com/apache/spark/pull/31461#issuecomment-775709570
> @jnh5y @viirya I got a good question from @marmbrus - can you support user-defined types by just defining an Encoder for it? so that it can work in a Dataset? This sounds a more tremendous change than the PR I proposed. I think `Encoder` is for converting JVM object to top-level row in Spark SQL. Currently `Encoder` is a pretty abstract trait without any defined APIs, excepts for returning its schema and class tag. Only one implementation of `Encoder` is `ExpressionEncoder`, and it looks more for internal usage as it is based on catalyst expressions. If using `Encoder` for user-defined types is the plan, we probably need to add public API for users to define `Encoder`? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
