[
https://issues.apache.org/jira/browse/BEAM-4454?focusedWorklogId=171559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171559
]
ASF GitHub Bot logged work on BEAM-4454:
----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Dec/18 12:58
Start Date: 03/Dec/18 12:58
Worklog Time Spent: 10m
Work Description: kanterov commented on a change in pull request #7181:
[BEAM-4454] Add more AVRO utilities to convert between Beam and Avro.
URL: https://github.com/apache/beam/pull/7181#discussion_r238257876
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
##########
@@ -198,50 +498,20 @@ public static Object convertAvroFieldStrict(
throw new IllegalArgumentException("Can't convert 'null' to
non-nullable field");
default:
- throw new AssertionError("Unexpected AVRO Schema.Type: " +
unwrapped.getType());
+ throw new AssertionError("Unexpected AVRO Schema.Type: " +
type.type.getType());
}
}
- @VisibleForTesting
- static org.apache.avro.Schema unwrapNullableSchema(org.apache.avro.Schema
avroSchema) {
- if (avroSchema.getType() == org.apache.avro.Schema.Type.UNION) {
- List<org.apache.avro.Schema> types = avroSchema.getTypes();
-
- // optional fields in AVRO have form of:
- // {"name": "foo", "type": ["null", "something"]}
-
- // don't need recursion because nested unions aren't supported in AVRO
- List<org.apache.avro.Schema> nonNullTypes =
- types
- .stream()
- .filter(x -> x.getType() != org.apache.avro.Schema.Type.NULL)
- .collect(Collectors.toList());
-
- if (nonNullTypes.size() == types.size()) {
- // union without `null`, keep as is
- return avroSchema;
- } else if (nonNullTypes.size() > 1) {
- return org.apache.avro.Schema.createUnion(nonNullTypes);
- } else if (nonNullTypes.size() == 1) {
- return nonNullTypes.get(0);
- } else { // nonNullTypes.size() == 0
- return avroSchema;
- }
- }
-
- return avroSchema;
- }
-
private static Object convertRecordStrict(GenericRecord record,
Schema.FieldType fieldType) {
checkTypeName(fieldType.getTypeName(), Schema.TypeName.ROW, "record");
- return toRowStrict(record, fieldType.getRowSchema());
+ return toBeamRowStrict(record, fieldType.getRowSchema());
}
private static Object convertBytesStrict(ByteBuffer bb, Schema.FieldType
fieldType) {
checkTypeName(fieldType.getTypeName(), Schema.TypeName.BYTES, "bytes");
byte[] bytes = new byte[bb.remaining()];
- bb.get(bytes);
+ bb.duplicate().get(bytes);
Review comment:
Good catch! As I understand, `.get(byte[])` will mutate ByteBuffer
`position` and it will fail on the second attempt, unless we duplicate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 171559)
Time Spent: 3h 10m (was: 3h)
> Provide automatic schema registration for AVROs
> -----------------------------------------------
>
> Key: BEAM-4454
> URL: https://issues.apache.org/jira/browse/BEAM-4454
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
> Reporter: Reuven Lax
> Assignee: Reuven Lax
> Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Need to make sure this is a compatible change
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
