I had asked this in a github issue but I'm reposting here to try and get an
answer from a wider audience.

Has any thought gone into how kafka-connect-jdbc will be impacted by SQL
transactions committing IDs and timestamps out-of-order?  Let me give an
example with two connections.

1: begin transaction
1: insert (get id 1)
2: begin transaction
2: insert (get id 2)
2: commit (recording id 2)
kafka-connect-jdbc runs and thinks it has handled everything through id 2
1: commit (recording id 1)

This would result in kafka-connect-jdbc missing id 1. The same thing could
happen with timestamps. I've read through some of the kafka-connect-jdbc
code and I think it may be susceptible to this problem, but I haven't run
it or verified that it would be an issue. Has this come up before? Are
there plans to deal with this situation?

Obviously something like bottled-water for postgresql would handle this
nicely as it would get the changes once they're committed.


Thanks for any insight,

Mark.


Original github issue:
https://github.com/confluentinc/kafka-connect-jdbc/issues/27

Reply via email to