[ https://issues.apache.org/jira/browse/BEAM-4454?focusedWorklogId=171384&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171384 ]
ASF GitHub Bot logged work on BEAM-4454: ---------------------------------------- Author: ASF GitHub Bot Created on: 02/Dec/18 16:55 Start Date: 02/Dec/18 16:55 Worklog Time Spent: 10m Work Description: kanterov commented on a change in pull request #7181: [BEAM-4454] Add more AVRO utilities to convert between Beam and Avro. URL: https://github.com/apache/beam/pull/7181#discussion_r238104103 ########## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java ########## @@ -111,28 +234,143 @@ public static Schema toSchema(@Nonnull org.apache.avro.Schema schema) { default: throw new AssertionError("Unexpected AVRO Schema.Type: " + avroSchema.getType()); } + fieldType = fieldType.withNullable(type.nullable); + return fieldType; } - /** - * Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during - * conversion. - */ - public static Row toRowStrict(@Nonnull GenericRecord record, @Nonnull Schema schema) { - Row.Builder builder = Row.withSchema(schema); - org.apache.avro.Schema avroSchema = record.getSchema(); + private static org.apache.avro.Schema getFieldSchema(Schema.FieldType fieldType) { + org.apache.avro.Schema baseType; + switch (fieldType.getTypeName()) { + case BYTE: + case INT16: + case INT32: + baseType = org.apache.avro.Schema.create(Type.INT); + break; - for (Schema.Field field : schema.getFields()) { - Object value = record.get(field.getName()); - org.apache.avro.Schema fieldAvroSchema = avroSchema.getField(field.getName()).schema(); + case INT64: + baseType = org.apache.avro.Schema.create(Type.LONG); + break; - if (value == null) { - builder.addValue(null); - } else { - builder.addValue(convertAvroFieldStrict(value, fieldAvroSchema, field.getType())); - } + case DECIMAL: + // TODO: Use a string representation. + throw new IllegalArgumentException("Avro does not support decimal types."); Review comment: Since AVRO 1.8 there are [logical types](https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types). Out of the box, there is support for: `decimal`, `date`, `time` and `timestamp` (both ms and us). Logical types already supported by codegen in avro-compiler. Should we use them instead of strings? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 171384) Time Spent: 50m (was: 40m) > Provide automatic schema registration for AVROs > ----------------------------------------------- > > Key: BEAM-4454 > URL: https://issues.apache.org/jira/browse/BEAM-4454 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core > Reporter: Reuven Lax > Assignee: Reuven Lax > Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Need to make sure this is a compatible change -- This message was sent by Atlassian JIRA (v7.6.3#76005)