Re: getting encoder implicits to be more accurate

s and >> provide your own, just like any type class. >> >> https://github.com/upio/spark-sql-formats >> >> >> *From:* Michael Armbrust <mich...@databricks.com> >> *Date:* October 26, 2016 at 12:50:23 PM PDT >> *To:* Koert Kuipers <ko...@tresata.

Re: getting encoder implicits to be more accurate

so > entirely pluggable meaning you can bypass the default implementations and > provide your own, just like any type class. > > https://github.com/upio/spark-sql-formats > > > *From:* Michael Armbrust <mich...@databricks.com> > *Date:* October 26, 2016 at 12:50:23 PM

Re: getting encoder implicits to be more accurate

t; >> >> *From:* Michael Armbrust <mich...@databricks.com> >> *Date:* October 26, 2016 at 12:50:23 PM PDT >> *To:* Koert Kuipers <ko...@tresata.com> >> *Cc:* Ryan Blue <rb...@netflix.com>, "dev@spark.apache.org" < >> dev@spark.apache

Re: getting encoder implicits to be more accurate

; *To:* Koert Kuipers <ko...@tresata.com> >> *Cc:* Ryan Blue <rb...@netflix.com>, "dev@spark.apache.org" < >> dev@spark.apache.org> >> *Subject:* *Re: getting encoder implicits to be more accurate* >> >> Sorry, I realize that set is only one ex

Re: getting encoder implicits to be more accurate

* Ryan Blue <rb...@netflix.com>, "dev@spark.apache.org" < > dev@spark.apache.org> > *Subject:* *Re: getting encoder implicits to be more accurate* > > Sorry, I realize that set is only one example here, but I don't think that > making the type of the implicit more narro

Re: getting encoder implicits to be more accurate

, 2016 at 12:50:23 PM PDT *To:* Koert Kuipers <ko...@tresata.com> *Cc:* Ryan Blue <rb...@netflix.com>, "dev@spark.apache.org" < dev@spark.apache.org> *Subject:* *Re: getting encoder implicits to be more accurate* Sorry, I realize that set is only one example here, but

Re: getting encoder implicits to be more accurate

Awesome, this is a great idea. I opened SPARK-18122 . On Wed, Oct 26, 2016 at 2:11 PM, Koert Kuipers wrote: > if kryo could transparently be used for subtrees without narrowing the > implicit that would be great > > On Wed,

Re: getting encoder implicits to be more accurate

if kryo could transparently be used for subtrees without narrowing the implicit that would be great On Wed, Oct 26, 2016 at 5:10 PM, Koert Kuipers wrote: > i use kryo for the whole thing currently > > it would be better to use it for the subtree > > On Wed, Oct 26, 2016 at

Re: getting encoder implicits to be more accurate

i use kryo for the whole thing currently it would be better to use it for the subtree On Wed, Oct 26, 2016 at 5:06 PM, Michael Armbrust wrote: > You use kryo encoder for the whole thing? Or just the subtree that we > don't have specific encoders for? > > Also, I'm

Re: getting encoder implicits to be more accurate

You use kryo encoder for the whole thing? Or just the subtree that we don't have specific encoders for? Also, I'm saying I like the idea of having a kryo fallback. I don't see the point of narrowing the the definition of the implicit. On Wed, Oct 26, 2016 at 1:07 PM, Koert Kuipers

Re: getting encoder implicits to be more accurate

for example (the log shows when it creates a kryo encoder): scala> implicitly[EncoderEvidence[Option[Seq[String.encoder res5: org.apache.spark.sql.Encoder[Option[Seq[String]]] = class[value[0]: array] scala> implicitly[EncoderEvidence[Option[Set[String.encoder dataframe.EncoderEvidence$:

Re: getting encoder implicits to be more accurate

why would generating implicits for ProductN where you also require the elements in the Product to have an expression encoder not work? we do this. and then we have a generic fallback where it produces a kryo encoder. for us the result is that say an implicit for Seq[(Int, Seq[(String, Int)])]

Re: getting encoder implicits to be more accurate

Sorry, I realize that set is only one example here, but I don't think that making the type of the implicit more narrow to include only ProductN or something eliminates the issue. Even with that change, we will fail to generate an encoder with the same error if you, for example, have a field of

Re: getting encoder implicits to be more accurate

yup, it doesnt really solve the underlying issue. we fixed it internally by having our own typeclass that produces encoders and that does check the contents of the products, but we did this by simply supporting Tuple1 - Tuple22 and Option explicitly, and not supporting Product, since we dont have

Re: getting encoder implicits to be more accurate

Isn't the problem that Option is a Product and the class it contains isn't checked? Adding support for Set fixes the example, but the problem would happen with any class there isn't an encoder for, right? On Wed, Oct 26, 2016 at 11:18 AM, Michael Armbrust wrote: > Hmm,

Re: getting encoder implicits to be more accurate

Hmm, that is unfortunate. Maybe the best solution is to add support for sets? I don't think that would be super hard. On Tue, Oct 25, 2016 at 8:52 PM, Koert Kuipers wrote: > i am trying to use encoders as a typeclass where if it fails to find an > ExpressionEncoder it falls