Re: Kafka Connect - how to deal with multiple formats in Kafka?

2016-08-22 Thread Michael Sklyar
Thank you, Glad to see it is addressed. On Mon, Aug 22, 2016 at 8:36 PM, Dustin Cote wrote: > Hi Michael, > > You'd probably be interested in the discussion for this KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 75+-+Add+per-connector+Converters > > For now, you'd have to run d

Kafka Connect - how to deal with multiple formats in Kafka?

2016-08-22 Thread Michael Sklyar
I am looking into Kafka Connect and Confluent HDFSSinkConnector. The goal is to save data from various topics to HDFS. We have at least two different formats of the data in Kafka - raw data (JSON) - that we want to save as SequenceFile and normalized data (Protobuf) that we want to save as Parquet

Re: Kafka Connect - how to deal with multiple formats in Kafka?

2016-08-22 Thread Dustin Cote
Hi Michael, You'd probably be interested in the discussion for this KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP- 75+-+Add+per-connector+Converters For now, you'd have to run different Connect instances, but KIP-75 plans to let you have control over converters at a connector level.