You don't need compiler time macros for this, you can do it quite easily using shapeless. I've been playing with a project which borrows ideas from spray-json and spray-json-shapeless to implement Row marshalling for arbitrary case classes. It's checked and generated at compile time, supports arbitrary/nested case classes, and allows custom types. It is also entirely pluggable meaning you can bypass the default implementations and provide your own, just like any type class.
https://github.com/upio/spark-sql-formats *From:* Michael Armbrust <mich...@databricks.com> *Date:* October 26, 2016 at 12:50:23 PM PDT *To:* Koert Kuipers <ko...@tresata.com> *Cc:* Ryan Blue <rb...@netflix.com>, "dev@spark.apache.org" < dev@spark.apache.org> *Subject:* *Re: getting encoder implicits to be more accurate* Sorry, I realize that set is only one example here, but I don't think that making the type of the implicit more narrow to include only ProductN or something eliminates the issue. Even with that change, we will fail to generate an encoder with the same error if you, for example, have a field of your case class that is an unsupported type. Short of changing this to compile-time macros, I think we are stuck with this class of errors at runtime. The simplest solution seems to be to expand the set of thing we can handle as much as possible and allow users to turn on a kryo fallback for expression encoders. I'd be hesitant to make this the default though, as behavior would change with each release that adds support for more types. I would be very supportive of making this fallback a built-in option though. On Wed, Oct 26, 2016 at 11:47 AM, Koert Kuipers <ko...@tresata.com> wrote: yup, it doesnt really solve the underlying issue. we fixed it internally by having our own typeclass that produces encoders and that does check the contents of the products, but we did this by simply supporting Tuple1 - Tuple22 and Option explicitly, and not supporting Product, since we dont have a need for case classes if case classes extended ProductN (which they will i think in scala 2.12?) then we could drop Product and support Product1 - Product22 and Option explicitly while checking the classes they contain. that would be the cleanest. On Wed, Oct 26, 2016 at 2:33 PM, Ryan Blue <rb...@netflix.com> wrote: Isn't the problem that Option is a Product and the class it contains isn't checked? Adding support for Set fixes the example, but the problem would happen with any class there isn't an encoder for, right? On Wed, Oct 26, 2016 at 11:18 AM, Michael Armbrust <mich...@databricks.com> wrote: Hmm, that is unfortunate. Maybe the best solution is to add support for sets? I don't think that would be super hard. On Tue, Oct 25, 2016 at 8:52 PM, Koert Kuipers <ko...@tresata.com> wrote: i am trying to use encoders as a typeclass where if it fails to find an ExpressionEncoder it falls back to KryoEncoder. the issue seems to be that ExpressionEncoder claims a little more than it can handle here: implicit def newProductEncoder[T <: Product : TypeTag]: Encoder[T] = Encoders.product[T] this "claims" to handle for example Option[Set[Int]], but it really cannot handle Set so it leads to a runtime exception. would it be useful to make this a little more specific? i guess the challenge is going to be case classes which unfortunately dont extend Product1, Product2, etc. -- Ryan Blue Software Engineer Netflix
