Hi, Xiangdong, you can absolutely join the discussion, please take your time and am looking forward to hearing from you.
Otto, yes it would be a breaking change. Changing the names of the fields in the schema we can look at, I know some of them are keywords in the Avro schema though. Chris, reaching out to the kafka and iot mailing lists is a great idea. Ben. On Sat, Nov 7, 2020 at 8:25 PM Xiangdong Huang <[email protected]> wrote: > Hi, > > Though I am not from Kafka project, maybe I can join the discussion > as I have some experiences and usually we (mean, the database) consumes > data from Kafka. > > I am reading the current Kafka adapter's implementation, > and will be back soon :D > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Christofer Dutz <[email protected]> 于2020年11月8日周日 上午5:26写道： > > > Hi Ben, > > > > I have to admit that even if I created most of the connector, I have > > almost no experience with Kafka. > > I don't even understand the schema topic ;-) > > > > But it does make sense from your email and I can imagine that the current > > way it's implemented could be an issue. > > I have no objections to changing this. > > > > Perhaps it would be cool to involve some people from the Kafka project? > > Perhaps callout on the [email protected] mailing-list too? (I'd probably do > > both, if I were you) > > > > Chris > > > > > > > > Am 07.11.20, 14:31 schrieb "Otto Fowler" <[email protected]>: > > > > Is it a breaking change? Is anyone using it? > > I would be fine with changing it, but we would need to be clear on > > those > > things. > > Also, I think the names should reflect what the project uses, > > > > plcfields not fields etc > > > > From: Ben Hutcheson <[email protected]> <[email protected]> > > Reply: [email protected] <[email protected]> < > > [email protected]> > > Date: November 7, 2020 at 06:10:42 > > To: [email protected] <[email protected]> < > [email protected]> > > Subject: [DISCUSS] Changing Schema for Kafka Source Connector > > > > Hi, > > > > Putting together a doc for the schema for the source connector for > > Kafka it > > appears that the existing schema is something like this:- > > > > { > > "type": "record", > > "name": "source-connector", > > "namespace": "com.apache.plc4x.kafka.config", > > "fields": [ > > { > > "name": "running", > > "type": "int16" > > }, > > { > > "name": "conveyorLeft", > > "type": "int16" > > }, > > .... > > ] > > } > > > > In which the schema needs to be modified depending on what tags are > > being > > collected. > > > > If we change it so that the tags are included in an array of > name/value > > pairs then we wouldn't have to modify the schema when tags are > > added/deleted. The new schema would be something like this. > > > > { > > "type": "record", > > "name": "source-connector", > > "namespace": "com.apache.plc4x.kafka.config", > > "fields": [ > > { > > "name": "tag", > > "type":{ > > "type": "array", > > "items":{ > > "name":"tag", > > "type":"record", > > "fields":[ > > {"name":"name", "type":"string"}, > > {"name":"value", "type":"string"} <- With this > > eventually being a union of different types. > > ] > > } > > } > > } > > { > > "name": "timestamp", > > "type": "string" > > } > > ] > > } > > > > What are your thoughts on changing this? It would allow us not to > have > > to > > send the schema within each packet. > > > > It does increase the packet size for the specific case that the tags > > will > > never change and the schema isn't being included in each packet, but > I > > think this would be few and far between. > > > > Ben > > > > >
