Hi Sagar, great that you managed to have a look ... I'll try to answer your questions. (I like to answer them postfix as whenever emails are sort of answered in-line, they are extremely hard to read and follow on mobile email clients __ )
First of all I created the original plugin via the archetype for kafka-connect plugins. The next thing I did, was to have a look at the code of the JDBC Kafka Connect plugin (as you might have guessed) as I thought that it would have similar structure as we do. Unfortunately I think the JDBC plugin is far more complex than the plc4x connector will have to be. I sort of picked some of the things I liked with the archetype and some I liked with the jdbc ... if there was a third, even cooler option ... I will definitely have missed that. So if you think there is a thing worth changing ... you can change anything you like. 1) The code of the jdbc plugin showed such a while(true) loop, however I think this was because the jdbc query could return a lot of rows and hereby Kafka events. In our case we have one request and get one response. The code in my example directly calls "get()" on the request and is hereby blocking. I don't know if this is good, but from reading the jdbc example, this should be blocking too ... So the PlcReaders read() method returns a completable future ... this could be completed asynchronously and the callback could fire the kafka events, but I didn't know if this was ok with kafka. If it is possible, please have a look at this example code: https://github.com/apache/incubator-plc4x/blob/master/plc4j/protocols/s7/src/test/java/org/apache/plc4x/java/s7/S7PlcReaderSample.java It demonstrates with comments the different usage types. While at it ... is there also an option for a Kafka connector that is able to push data? So if an incoming event arrives, this is automatically pushed without a fixed polling interval? 2) I have absolutely no idea as I am not quite familiar with the concepts inside kafka. What I do know is that probably the partition-key should be based upon the connection url. The problem is, that with kafka I could have 1000 nodes connecting to one PLC. While Kafka wouldn't have problems with that, the PLCs have very limited resources. So as far as I decoded the responses of my Siemens S7 1200 it can handle up to 20 connections (Usually a control-system already consuming 2-3 of them) ... I think it would be ideal, if on one Kafka node (or partition) there would be one PlcConnection ... this connection should then be shared among all requests to a PLC with a shared connection url (I hope I'm not writing nonsense). So if a workerTask is responsible for managing all request to one partition, then I'd say it should be 1 ... otherwise the number could be bigger. If it makes things easier, I'm absolutely fine with using those ConnectorUtils Regarding the connector offsets ... are you referring to that counter Kafka uses to let the clients know the sequence of events and which they use to sort of say: "Hi, I have number 237367 of topic 'ABC', plese continue" ... is that what you are referring to? If it is, well ... I have to admit ... I don't know ... ok ... if it isn't then probably also ;-) How do other plugins do this? 3) Well I guess both options would be cool ... JSON is definitely simpler, but for high volume transports the binary counterparts definitely are worth consideration. Currently PLC4X tries to deliver what you request, but that's actually something we're currently discussing on refactoring. But for the moment - as shown in the example code I referenced a few lines above - you do a TypedRequest and for example ask for an Integer, then you will receive an array (probably of size 1) of Integers. 4) Well I agree ... well at least I can't even say that I make a secret about where I stole things from ;-) If I can be of any assistance ... just ask. Thanks for taking the time. Chris Am 22.08.18, 17:55 schrieb "Sagar" <sagarmeansoc...@gmail.com>: Hi All, I was going through the K-Connect stubs created by Chris in the kafka feature branch. Some of the findings I found are here(let me know if they are valid or not): 1) https://github.com/apache/incubator-plc4x/blob/feature/apache-kafka/integrations/apache-kafka/src/main/java/org/apache/plc4x/kafka/source/Plc4xSourceTask.java#L98 Should this block of code be within an infinite loop like while(true)? I am not exactly sure of the semantics of the PlcReader hence asking this question. 2) Another question is, what are the maxTasks that we envision here? https://github.com/apache/incubator-plc4x/blob/feature/apache-kafka/integrations/apache-kafka/src/main/java/org/apache/plc4x/kafka/Plc4xSourceConnector.java#L46 Also, as part of documentation, there's a utility called ConnectorUtils which typically should be used to create the configs(not a hard and fast rule though): https://docs.confluent.io/current/connect/javadocs/index.html?org/apache/kafka/connect/util/ConnectorUtils.html If we go that route, then we also need to specify how the offsets would be stored in the offsets topic(by using the task name). So, if it can be figured out as to how would the connectors be setup, then that'll be helpful. 3) While building the SourceRecord -> https://github.com/apache/incubator-plc4x/blob/feature/apache-kafka/integrations/apache-kafka/src/main/java/org/apache/plc4x/kafka/source/Plc4xSourceTask.java#L109 , we would also need some DataConverter layer to have them mapped to the connect types. Also, which message types would be supported? Json or binary protocols like Avro/protobuf etc or some other protocols? Those things might also need to be factored in. 4) Lastly, need to remove the JdbcSourceTask from the catch block here :) -> https://github.com/apache/incubator-plc4x/blob/feature/apache-kafka/integrations/apache-kafka/src/main/java/org/apache/plc4x/kafka/source/Plc4xSourceTask.java#L67 Thanks! Sagar.
