[ https://issues.apache.org/jira/browse/SPARK-27388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16817089#comment-16817089 ]
Taoufik DACHRAOUI edited comment on SPARK-27388 at 4/13/19 10:05 PM: --------------------------------------------------------------------- Another solution was created in [https://github.com/apache/spark/pull/24367] was (Author: mazeboard): Currently we modified JavaTypeInference to be able to create encoders for Avro objects; we have now two solutions, the one in this PR and another one using Encoders.bean (fewer code changes); which one is better? how to proceed? we should create another PR for the Encoders.bean solution? The test we used for Encoders.bean is as follows: {code:java} implicit val avroExampleEncoder = Encoders.bean[AvroExample1](classOf[AvroExample1]).asInstanceOf[ExpressionEncoder[AvroExample1]] val input: AvroExample1 = AvroExample1.newBuilder() .setMyarray(List("Foo", "Bar").asJava) .setMyboolean(true) .setMybytes(java.nio.ByteBuffer.wrap("MyBytes".getBytes())) .setMydouble(2.5) .setMyfixed(new Magic("magic".getBytes)) .setMyfloat(25.0F) .setMyint(100) .setMylong(10L) .setMystring("hello") .setMymap(Map( "foo" -> new java.lang.Integer(1), "bar" -> new java.lang.Integer(2)).asJava) .setMymoney(Money.newBuilder().setAmount(100.0F).setCurrency(Currency.EUR).build()) .build() val row: InternalRow = avroExampleEncoder.toRow(input) val output: AvroExample1 = avroExampleEncoder.resolveAndBind().fromRow(row) val ds: Dataset[AvroExample1] = List(input).toDS() println(ds.schema) println(ds.collect().toList) ds.write.format("avro").save("example1") val fooDF = spark.read.format("avro").load("example1") val fooDS = fooDF.as[AvroExample1] println(fooDS.collect().toList) {code} The change is made in the branch [https://github.com/mazeboard/spark/tree/bean-encoder] *with 105 additions and 39 deletions. (excluding the code for tests)* > expression encoder for avro objects > ----------------------------------- > > Key: SPARK-27388 > URL: https://issues.apache.org/jira/browse/SPARK-27388 > Project: Spark > Issue Type: New Feature > Components: SQL > Affects Versions: 2.4.1 > Reporter: Taoufik DACHRAOUI > Priority: Major > > *What changes were proposed in this pull request?* > The PR adds support for bean objects, java.util.List, java.util.Map, > java.nio.ByteBuffer and java enums to ScalaReflection; unlike the existing > javaBean Encoder, properties can be named without the set/get prefix (this is > one of the key points that allows the encoding of Avro Fixed types. I > believe, the other key point is that the addition must be in ScalaReflection). > Reminder of Avro types: > * primitive types: null, boolean, int, long, float, double, bytes, string > * complex types: Records, Enums, Arrays, Maps, Unions, Fixed > This PR supports simple unions (having a null type and a non-null type) but > not complex unions for the simple reason that the Avro compiler will generate > java code with type Object for all complex unions, and fields with simple > unions will be typed as the non-null type of the union. > > *How was this patch tested?* > currently only 1 encodeDecodeTest was added to ExpressionEncoderSuite; the > test uses the following avro schema: > {code:java} > {"namespace": "org.apache.spark.sql.catalyst.encoders", "type": "record", > "name": "AvroExample1", > "fields": [ > > {"name":"mymoney","type":["null",{"type":"record","name":"Money","namespace":"org.apache.spark.sql.catalyst.encoders","fields":[ > {"name":"amount","type":"float","default":0}, > > {"name":"currency","type":{"type":"enum","name":"Currency","symbols":["EUR","USD","BRL"]},"default":"EUR"}]}], > "default":null}, > {"name": "myfloat", "type": "float"}, > {"name": "mylong", "type": "long"}, > {"name": "myint", "type": "int"}, > {"name": "mydouble", "type": "double"}, > {"name": "myboolean", "type": "boolean"}, > {"name": "mystring", "type": "string"}, > {"name": "mybytes", "type": "bytes"}, > {"name": "myfixed", "type": {"type": "fixed", "name": "Magic", "size": 4}}, > {"name": "myarray", "type": {"type": "array", "items": "string"}}, > {"name": "mymap", "type": {"type": "map", "values": "int"}} > ] }{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org