benkonz commented on code in PR #30770:
URL: https://github.com/apache/beam/pull/30770#discussion_r1567714079
##########
sdks/python/apache_beam/io/avroio.py:
##########
@@ -649,7 +684,11 @@ def beam_schema_to_avro_schema(
def beam_type_to_avro_type(beam_type: schema_pb2.FieldType) -> _AvroSchemaType:
type_info = beam_type.WhichOneof("type_info")
if type_info == "atomic_type":
- return {'type': BEAM_PRIMITIVES_TO_AVRO_PRIMITIVES[beam_type.atomic_type]}
+ avro_primitive = BEAM_PRIMITIVES_TO_AVRO_PRIMITIVES[beam_type.atomic_type]
+ if beam_type.nullable:
+ return ['null', avro_primitive]
Review Comment:
I spent some more time thinking about this, and the conclusion I came up
with was: always return a dictionary in `beam_type_to_avro_type`, when we are
generating the avro type for a beam row_type, if the field we are mapping is an
atomic_type, unnest the field from `{type: type: { [int, null ] } }` to `{type:
[int, null ] }`, but for complex types generate the field type the same way we
previously were. That way we shouldn't introduce any new bugs, but should still
be able to convert from avro schemas to beam schemas and back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
