John Hofman created KAFKA-3910: ---------------------------------- Summary: Cyclic schema support in ConnectSchema and SchemaBuilder Key: KAFKA-3910 URL: https://issues.apache.org/jira/browse/KAFKA-3910 Project: Kafka Issue Type: Improvement Components: KafkaConnect Affects Versions: 0.10.0.0 Reporter: John Hofman Assignee: Ewen Cheslack-Postava Priority: Blocker
Cyclic schema's are not supported by ConnectSchema or SchemaBuilder. Subsequently the AvroConverter (confluentinc/schema-registry) hits a stack overflow when converting a cyclic avro schema, e.g: {code} {"type":"record", "name":"list","fields":[{"name":"value","type":"int"},{"name":"next","type":["null","list"]}]} {code} This is a blocking issue for all connectors running on the connect framework and with data containing cyclic references. The AvroConverter cannot support cyclic schema's until the underlying ConnectSchema and SchemaBuilder do. To reproduce the stack-overflow (Confluent-3.0.0): Produce some cyclic data: {code} bin/kafka-avro-console-producer --broker-list localhost:9092 --topic test --property value.schema='{"type":"record", "name":"list","fields":[{"name":"value","type":"int"},{"name":"next","type":["null","list"]}]}' {"value":1,"next":null} {"value":1,"next":{"list":{"value":2,"next":null}}} {code} Then try to consume it with connect: {code:title=connect-console-sink.properties} name=local-console-sink connector.class=org.apache.kafka.connect.file.FileStreamSinkConnector tasks.max=1 topics=test {code} {code} ./bin/connect-standalone ./etc/schema-registry/connect-avro-standalone.properties connect-console-sink.properties … start up logging … java.lang.StackOverflowError at org.apache.avro.JsonProperties.getJsonProp(JsonProperties.java:54) at org.apache.avro.JsonProperties.getProp(JsonProperties.java:45) at io.confluent.connect.avro.AvroData.toConnectSchema(AvroData.java:1055) at io.confluent.connect.avro.AvroData.toConnectSchema(AvroData.java:1103) at io.confluent.connect.avro.AvroData.toConnectSchema(AvroData.java:1137) at io.confluent.connect.avro.AvroData.toConnectSchema(AvroData.java:1103) at io.confluent.connect.avro.AvroData.toConnectSchema(AvroData.java:1137) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)