Thanks Ewan,
The second request was made by me directly. I'm trying to add this
functionality into my .Net application. The library I'm using doesn't have an
implementation of the AvroSeriazlizer that interacts with the schema registry.
I've now added in code to make to POST to /subjects/<topic>-value with the
schema. Something I noticed is that I was using schema like this:
{
"subject": "AuditHdfsTest5-value",
"version": 1,
"id": 5,
"schema":
"{\"type\":\"record\",\"name\":\"GenericAuditRecord\",\"namespace\":\"audit\",\"fields\":[{\"name\":\"xml\",\"type\":[\"string\",\"null\"]}]}"
}
When the connector got a message and did a lookup it didn't have the
"namespace" field and the lookup failed. I then posted a new version of the
schema without the "namespace" field and it worked.
-Dave
Dave Tauzell | Senior Software Engineer | Surescripts
O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com
Connect with us: Twitter I LinkedIn I Facebook I YouTube
-----Original Message-----
From: Ewen Cheslack-Postava [mailto:e...@confluent.io]
Sent: Tuesday, June 14, 2016 6:59 PM
To: users@kafka.apache.org
Subject: Re: Kafka Connect HdfsSink and the Schema Registry
On Tue, Jun 14, 2016 at 8:08 AM, Tauzell, Dave <dave.tauz...@surescripts.com
> wrote:
> I have been able to get my C# client to put avro records to a Kafka
> topic and have the HdfsSink read and save them in files. I am
> confused about interaction with the registry. The kafka message
> contains a schema id an I see the connector look that up in the
> registry. Then it also looks up a subject which is <topic>-value.
>
> What is the relationship between the passed schema id and the subject
> which is derived from the topic name?
>
The HDFS connector doesn't work directly with the schema registry, the
AvroConverter does. I'm not sure what the second request you're seeing is
-- normally it would only look up the schema ID in order to get the schema.
Where are you seeing the second request, and can you include some logs? I can't
think of any other requests the AvroConverter would be making just for
deserialization.
The subject names are generating in the serializer as <topic>-key and
<topic>-value and this is just the standardized approach Confluent's
serializers use. The ID will have been registered under that subject.
-Ewen
>
> -Dave
>
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of
> the individual or entity to whom they are addressed. If you have
> received this e-mail in error, please notify the sender by reply
> e-mail immediately and destroy all copies of the e-mail and any attachments.
>
--
Thanks,
Ewen
This e-mail and any files transmitted with it are confidential, may contain
sensitive information, and are intended solely for the use of the individual or
entity to whom they are addressed. If you have received this e-mail in error,
please notify the sender by reply e-mail immediately and destroy all copies of
the e-mail and any attachments.
