Re: Data in kafka topic in Json format
You have shared the Kafka connect properties but not the source connector config. Which source connector are you using? Does it override the default settings you provided? Are you running the connector in standalone mode or distributed mode? Also what are you using to consume the messages and see the message format? -hans > On Jun 2, 2017, at 9:10 AM, Mina Aslani wrote: > > Hi Hans, > > Thank you for your quick response, appreciate it. > > In *kafka-connect* docker, I see below settings in > *kafka-connect.properties* file in *kafka-connect *directory: > > key.converter.schemas.enable=false > key.converter.schema.registry.url=http://kafka-schema-registry: > value.converter.schema.registry.url=http://kafka-schema-registry: > > key.converter=org.apache.kafka.connect.json.JsonConverter > value.converter=org.apache.kafka.connect.json.JsonConverter > > internal.value.converter.schemas.enable=false > rest.advertised.host.name=kafka-connect > internal.value.converter=org.apache.kafka.connect.json.JsonConverter > internal.key.converter=org.apache.kafka.connect.json.JsonConverter > And the settings in *schema-registry *directory of *kafka-connect *docker > are as > > https://github.com/confluentinc/schema-registry/tree/master/config > > Should I consider any other settings for *kafka-connect* or > *schema-registry* to get the real json object NOT string > formatted/stringified json which has extra "\" and is not json any more? > > Best regards, > Mina > > On Fri, Jun 2, 2017 at 11:18 AM, Hans Jespersen wrote: > >> >> My earlier comment still applies but in Kafka Connect the equivalent of a >> serializer/deserializer (serdes) is called a “converter”. >> Check which converter you have configured for your source connector and if >> it is overriding whatever the default converter is configured for the >> connect worker it is running in. >> >> -hans >> >> >> >> >>> On Jun 2, 2017, at 8:12 AM, Mina Aslani wrote: >>> >>> Hi, >>> >>> I would like to add that I use kafka-connect and schema-registery >> version ` >>> 3.2.1-6`. >>> >>> Best regards, >>> Mina >>> >>> On Fri, Jun 2, 2017 at 10:59 AM, Mina Aslani >> wrote: >>> Hi. Is there any way that I get the data into a Kafka topic in Json format? The source that I ingest the data from have the data in Json format, however when I look that data in the kafka topic, schema and payload >> fields are added and data is not in json format. I want to avoid implementing a transformer to get the data from the >> topic and publishes Json in another one. Your input is appreciated. Best regards, Mina >> >>
Re: Data in kafka topic in Json format
Hi Hans, Thank you for your quick response, appreciate it. In *kafka-connect* docker, I see below settings in *kafka-connect.properties* file in *kafka-connect *directory: key.converter.schemas.enable=false key.converter.schema.registry.url=http://kafka-schema-registry: value.converter.schema.registry.url=http://kafka-schema-registry: key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter.schemas.enable=false rest.advertised.host.name=kafka-connect internal.value.converter=org.apache.kafka.connect.json.JsonConverter internal.key.converter=org.apache.kafka.connect.json.JsonConverter And the settings in *schema-registry *directory of *kafka-connect *docker are as https://github.com/confluentinc/schema-registry/tree/master/config Should I consider any other settings for *kafka-connect* or *schema-registry* to get the real json object NOT string formatted/stringified json which has extra "\" and is not json any more? Best regards, Mina On Fri, Jun 2, 2017 at 11:18 AM, Hans Jespersen wrote: > > My earlier comment still applies but in Kafka Connect the equivalent of a > serializer/deserializer (serdes) is called a “converter”. > Check which converter you have configured for your source connector and if > it is overriding whatever the default converter is configured for the > connect worker it is running in. > > -hans > > > > > > On Jun 2, 2017, at 8:12 AM, Mina Aslani wrote: > > > > Hi, > > > > I would like to add that I use kafka-connect and schema-registery > version ` > > 3.2.1-6`. > > > > Best regards, > > Mina > > > > On Fri, Jun 2, 2017 at 10:59 AM, Mina Aslani > wrote: > > > >> Hi. > >> > >> Is there any way that I get the data into a Kafka topic in Json format? > >> The source that I ingest the data from have the data in Json format, > >> however when I look that data in the kafka topic, schema and payload > fields > >> are added and data is not in json format. > >> > >> I want to avoid implementing a transformer to get the data from the > topic > >> and publishes Json in another one. > >> > >> Your input is appreciated. > >> > >> Best regards, > >> Mina > >> > >
Re: Data in kafka topic in Json format
My earlier comment still applies but in Kafka Connect the equivalent of a serializer/deserializer (serdes) is called a “converter”. Check which converter you have configured for your source connector and if it is overriding whatever the default converter is configured for the connect worker it is running in. -hans > On Jun 2, 2017, at 8:12 AM, Mina Aslani wrote: > > Hi, > > I would like to add that I use kafka-connect and schema-registery version ` > 3.2.1-6`. > > Best regards, > Mina > > On Fri, Jun 2, 2017 at 10:59 AM, Mina Aslani wrote: > >> Hi. >> >> Is there any way that I get the data into a Kafka topic in Json format? >> The source that I ingest the data from have the data in Json format, >> however when I look that data in the kafka topic, schema and payload fields >> are added and data is not in json format. >> >> I want to avoid implementing a transformer to get the data from the topic >> and publishes Json in another one. >> >> Your input is appreciated. >> >> Best regards, >> Mina >>
Re: Data in kafka topic in Json format
Check which serializer you have configured in your producer. You are probably using an Avro serializer which will add the schema and modify the payload to avro data. You can use a String serializer or a ByteArray serializer and the data will either be Base64 encoded or not encoded at all. -hans > On Jun 2, 2017, at 7:59 AM, Mina Aslani wrote: > > Hi. > > Is there any way that I get the data into a Kafka topic in Json format? > The source that I ingest the data from have the data in Json format, > however when I look that data in the kafka topic, schema and payload fields > are added and data is not in json format. > > I want to avoid implementing a transformer to get the data from the topic > and publishes Json in another one. > > Your input is appreciated. > > Best regards, > Mina
Re: Data in kafka topic in Json format
Hi, I would like to add that I use kafka-connect and schema-registery version ` 3.2.1-6`. Best regards, Mina On Fri, Jun 2, 2017 at 10:59 AM, Mina Aslani wrote: > Hi. > > Is there any way that I get the data into a Kafka topic in Json format? > The source that I ingest the data from have the data in Json format, > however when I look that data in the kafka topic, schema and payload fields > are added and data is not in json format. > > I want to avoid implementing a transformer to get the data from the topic > and publishes Json in another one. > > Your input is appreciated. > > Best regards, > Mina >
Data in kafka topic in Json format
Hi. Is there any way that I get the data into a Kafka topic in Json format? The source that I ingest the data from have the data in Json format, however when I look that data in the kafka topic, schema and payload fields are added and data is not in json format. I want to avoid implementing a transformer to get the data from the topic and publishes Json in another one. Your input is appreciated. Best regards, Mina
