Hi all, I've created a small demo project to show the issue[1].
I've looked at the beam code and all the avro tests use avro 1.8... which is hardcoded to Joda, hence why all the tests pass. Avro changed to java.time (as most other recent java projects) after 1.8. Is there a plan for Beam to do the same? Does anyone use Avro with java.time instead of joda.time that could give me ideas how to make it work? Thank you, Cristian [1] https://github.com/zeidoo/beamjavadates-poc On Thu, Oct 14, 2021 at 5:35 PM Brian Hulette <bhule...@google.com> wrote: > +dev <d...@beam.apache.org> > > This sounds like a bug in the Avro schema mapping. I *think* the intention > is that we generate logic for converting Date to/from Instant when making a > getters for a RowWithGetters backed by Avro. > > Brian > > On Thu, Oct 14, 2021 at 4:43 AM Cristian Constantinescu <zei...@gmail.com> > wrote: > >> A little bit more color on this. >> >> That field is nested inside a avro record like so (not syntactically >> correct): >> { >> type: record, >> fields: [ >> { >> name: whatever, >> type: record { >> fields: [ >> { >> "name": "somedate", >> "type: {"type": "int", "logicalType": "date"} >> } >> ] >> } >> ] >> >> The outer layer record uses SchemaCoderHelper.coderForFieldType on all >> it's fields to create the coder. However, that method when called for the >> inner record field just returns a SchemaCoder.of(fieldType.getRowSchema()) >> which doesn't take into account LogicalTypes. >> >> I think that's where the problem is. If anyone who knows that code could >> have a look and let me know their thoughts, I can try to fix the issue if >> we agree that there is one. >> >> On Thu, Oct 14, 2021 at 7:12 AM Cristian Constantinescu <zei...@gmail.com> >> wrote: >> >>> Hi all, >>> >>> I have the following field in one of my avro schemas: >>> >>> { >>> "name": "somedate", >>> "type: {"type": "int", "logicalType": "date"} >>> } >>> >>> This generates a java.time.LocalDate field in the corresponding java >>> class (call it Foo). >>> >>> AvroUtils.toBeamSchema(FooClass.getSchema()) will return that field as >>> DATETIME in the Beam schema. because of AvroUtils.toFieldType (around line >>> 275, where TimestampMillis and Date are both stored as DATETIME). >>> >>> My problem is that in SchemaCoderHelpers, DATETIME is set to use >>> InstantCoder which expects a joda Instant as input not a LocalDate. So when >>> my PTransform returns Foo objects it crashes with "class >>> java.time.LocalDate cannot be cast to class org.joda.time.Instant..." when >>> trying to encode that field using InstantCoder.encode(). >>> >>> Is there a workaround for this issue? >>> >>> Thank you, >>> Cristian >>> >>> PS: I did search the mailing list and google, but didn't find anything >>> related except a thread on AvroCoder.JodaTimestampConversion, which I don't >>> think it applies to here. >>> >>
