[ https://issues.apache.org/jira/browse/AVRO-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-295: ------------------------------------- Attachment: AVRO-295-test-2.patch For the BinaryDecoder, flush() is a no-op. But, for BlockingBinaryEncoder and JsonEncoder flush does some useful things. Still, flushing in DatumWriter could be a performance problem for small objects. The DatumWriter doesn't own an encoder. The encoder gets passed in the write() method. So it doesn't make sense to add a flush() method in DatumWriter. Of course, the client can flush the encoder whenever it wants. test-2 patch demonstrates this. If we take this route, there is no fix required. Shall we resolve this JIRA as "Not a problem" then? Perhaps, I'll just check-in the test. > JsonEncoder is not flushed after writing using ReflectDatumWriter > ------------------------------------------------------------------ > > Key: AVRO-295 > URL: https://issues.apache.org/jira/browse/AVRO-295 > Project: Avro > Issue Type: Improvement > Components: java > Affects Versions: 1.3.0 > Reporter: Jonathan Hsieh > Assignee: Thiruvalluvan M. G. > Attachments: AVRO-295-test-2.patch, AVRO-295-test.patch, > AVRO-295.patch > > > JsonEncoder needs to be flushed otherwise data may be left in its buffers. > Ideally behavior should be the same regardless of what kind of Encoder passed > in. Here is some example code: > {code} > class A { > long timestamp; > } > public void testEventSchemaSerializeBinary() throws IOException { > A e = new A(); > e.timestamp = 1234; > ReflectData reflectData = ReflectData.get(); > Schema schm = reflectData.getSchema(A.class); > System.out.println(schm); > ReflectDatumWriter writer = new ReflectDatumWriter(schm); > ByteArrayOutputStream out = new ByteArrayOutputStream(); > Encoder json = new BinaryEncoder(out); > writer.write(e, json); // only one calls > byte[] bs = out.toByteArray(); > int len = bs.length; // length is 2, which is reasonable. > System.out.println("output size: " + len); > } > public void testSerializeJson() throws IOException { > A a = new A(); > a.timestamp = 1234; > ReflectData reflectData = ReflectData.get(); > Schema schm = reflectData.getSchema(A.class); > ReflectDatumWriter writer = new ReflectDatumWriter(schm); > ByteArrayOutputStream out = new ByteArrayOutputStream(); > JsonEncoder json = new JsonEncoder(schm, out); > writer.write(e, json); /// only one call > // did not flush > byte[] bs = out.toByteArray(); > int len = bs.length; // len == 0; this is unexpected! > System.out.println("output size: " + len); > > // flushed this time. this is a bit unwieldy > json.flush(); > bs = out.toByteArray(); > len = bs.length; // len == 18; this is better! > System.out.println("output size: " + len); > } > {code} > One way to deal with it is to have either all Encoders have flush method so > the DatumWriter can always flush it, and potentially add a flush method to > DatumWriter as well. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.