[ https://issues.apache.org/jira/browse/KAFKA-8315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew updated KAFKA-8315: -------------------------- Description: The problem we are experiencing is that we cannot reliably perform simple joins over pre-populated kafka topics. This seems more of a problem where one topic has records at less frequent record timestamp intervals that the other. An example of the issue is provided in this repository : [ [https://github.com/the4thamigo-uk/join-example]] Related slack conversations : [https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1556799561287300] [https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1557733979453900] 1) This issue was initially thought to be due to the inability to set the retention period for a join window via {{Materialized: i.e.}} The documentation says to use `Materialized` not `JoinWindows.until()` ([https://kafka.apache.org/22/javadoc/org/apache/kafka/streams/kstream/JoinWindows.html#until-long-]), but there is no where to pass a `Materialized` instance to the join operation, only to the group operation is supported it seems. This was considered to be a problem with the documentation not with the API and is addressed in [https://github.com/apache/kafka/pull/6664] 2) We then found an apparent issue in the code which would affect the partition that is selected to deliver the next record to the join. This would only be a problem for data that is out-of-order, and join-example uses data that is in order of timestamp in both topics. So this fix is thought not to affect join-example. This was considered to be an issue and is being addressed in [https://github.com/apache/kafka/pull/6719] 3) Further investigation using a crafted unit test seems to show that the partition-selection and ordering (PartitionGroup/RecordQueue) seems to work ok [https://github.com/the4thamigo-uk/kafka/commit/5121851491f2fd0471d8f3c49940175e23a26f1b] 4) the current assumption is that the issue is rooted in the way records are consumed from the topics : We have tried to set various options to suppress reads form the source topics but it doesnt seem to make any difference : https://github.com/the4thamigo-uk/join-example/commit/c674977fd0fdc689152695065d9277abea6bef63 was: The problem we are experiencing is that we cannot reliably perform simple joins over pre-populated kafka topics. This seems more of a problem where one topic has records at less frequent record timestamp intervals that the other. An example of the issue is provided in this repository : [ https://github.com/the4thamigo-uk/join-example] Related slack conversations : [https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1556799561287300] https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1557733979453900 1) This issue was initially thought to be due to the inability to set the retention period for a join window via {{Materialized: i.e.}} The documentation says to use `Materialized` not `JoinWindows.until()` ([https://kafka.apache.org/22/javadoc/org/apache/kafka/streams/kstream/JoinWindows.html#until-long-]), but there is no where to pass a `Materialized` instance to the join operation, only to the group operation is supported it seems. This was considered to be a problem with the documentation not with the API and is addressed in [https://github.com/apache/kafka/pull/6664] 2) We then found an apparent issue in the code which would affect the partition that is selected to deliver the next record to the join. This would only be a problem for data that is out-of-order, and join-example uses data that is in order of timestamp in both topics. So this fix is thought not to affect join-example. This was considered to be an issue and is being addressed in [https://github.com/apache/kafka/pull/6719] 3) Further investigation using a crafted unit test seems to show that the partition-selection and ordering (PartitionGroup/RecordQueue) seems to work ok [https://github.com/the4thamigo-uk/kafka/commit/5121851491f2fd0471d8f3c49940175e23a26f1b] 4) the current assumption is that the issue is rooted in the way records are consumed from the topics. > Historical join issues > ---------------------- > > Key: KAFKA-8315 > URL: https://issues.apache.org/jira/browse/KAFKA-8315 > Project: Kafka > Issue Type: Bug > Components: streams > Reporter: Andrew > Assignee: John Roesler > Priority: Major > Attachments: code.java > > > The problem we are experiencing is that we cannot reliably perform simple > joins over pre-populated kafka topics. This seems more of a problem where one > topic has records at less frequent record timestamp intervals that the other. > An example of the issue is provided in this repository : > [ > [https://github.com/the4thamigo-uk/join-example]] > > Related slack conversations : > [https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1556799561287300] > [https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1557733979453900] > > 1) This issue was initially thought to be due to the inability to set the > retention period for a join window via {{Materialized: i.e.}} > The documentation says to use `Materialized` not `JoinWindows.until()` > ([https://kafka.apache.org/22/javadoc/org/apache/kafka/streams/kstream/JoinWindows.html#until-long-]), > but there is no where to pass a `Materialized` instance to the join > operation, only to the group operation is supported it seems. > This was considered to be a problem with the documentation not with the API > and is addressed in [https://github.com/apache/kafka/pull/6664] > 2) We then found an apparent issue in the code which would affect the > partition that is selected to deliver the next record to the join. This would > only be a problem for data that is out-of-order, and join-example uses data > that is in order of timestamp in both topics. So this fix is thought not to > affect join-example. > This was considered to be an issue and is being addressed in > [https://github.com/apache/kafka/pull/6719] > 3) Further investigation using a crafted unit test seems to show that the > partition-selection and ordering (PartitionGroup/RecordQueue) seems to work ok > [https://github.com/the4thamigo-uk/kafka/commit/5121851491f2fd0471d8f3c49940175e23a26f1b] > 4) the current assumption is that the issue is rooted in the way records are > consumed from the topics : > We have tried to set various options to suppress reads form the source topics > but it doesnt seem to make any difference : > https://github.com/the4thamigo-uk/join-example/commit/c674977fd0fdc689152695065d9277abea6bef63 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)