[
https://issues.apache.org/jira/browse/KAFKA-8315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836735#comment-16836735
]
Andrew edited comment on KAFKA-8315 at 5/9/19 10:03 PM:
--------------------------------------------------------
Thanks [~ableegoldman] and [~vvcephei].
I have today tried to isolate the issue we are facing into a simple demo app. I
have two streams of data x and y, x is high frequency
(https://github.com/the4thamigo-uk/join-example/blob/master/data/x.dat) and y
is low frequency
https://github.com/the4thamigo-uk/join-example/blob/master/data/y.dat.
I set the join window sizes, and the grace here
https://github.com/the4thamigo-uk/join-example/blob/master/launch.sh#L9.
There is logging as records are read off the streams via a transformer here
https://github.com/the4thamigo-uk/join-example/blob/master/src/main/java/the4thamigouk/kafka/streams/joinexample/JoinerStreamProcessor.java#L54.
There is logging on the join here
https://github.com/the4thamigo-uk/join-example/blob/master/src/main/java/the4thamigouk/kafka/streams/joinexample/JoinerStreamProcessor.java#L93
What I see are joins only from here
https://github.com/the4thamigo-uk/join-example/blob/master/example.log#L1553
Also, I notice that all the x records are read first, then all the y records
https://github.com/the4thamigo-uk/join-example/blob/master/example.log#L451 and
https://github.com/the4thamigo-uk/join-example/blob/master/example.log#L1452
Am I doing something wrong here?
was (Author: the4thamigo_uk):
Thanks [~ableegoldman] and [~vvcephei].
I have today tried to isolate the issue we are facing into a simple demo app. I
have two streams of data x and y, x is high frequency
(https://github.com/the4thamigo-uk/join-example/blob/master/data/x.dat) and y
is low frequency
https://github.com/the4thamigo-uk/join-example/blob/master/data/y.dat.
I set the join window sizes, and the grace here
https://github.com/the4thamigo-uk/join-example/blob/master/launch.sh#L9.
There is logging as records are read off the streams via a transformer here
https://github.com/the4thamigo-uk/join-example/blob/master/src/main/java/the4thamigouk/kafka/streams/joinexample/JoinerStreamProcessor.java#L54.
There is logging on the join here
https://github.com/the4thamigo-uk/join-example/blob/master/src/main/java/the4thamigouk/kafka/streams/joinexample/JoinerStreamProcessor.java#L93
What I see are joins only from here
https://github.com/the4thamigo-uk/join-example/blob/master/example.log#L1553
Am I doing something wrong here?
> Cannot pass Materialized into a join operation - hence cant set retention
> period independent of grace
> -----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-8315
> URL: https://issues.apache.org/jira/browse/KAFKA-8315
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Andrew
> Assignee: John Roesler
> Priority: Major
> Attachments: code.java
>
>
> The documentation says to use `Materialized` not `JoinWindows.until()`
> ([https://kafka.apache.org/22/javadoc/org/apache/kafka/streams/kstream/JoinWindows.html#until-long-]),
> but there is no where to pass a `Materialized` instance to the join
> operation, only to the group operation is supported it seems.
>
> Slack conversation here :
> [https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1556799561287300]
> [Additional]
> From what I understand, the retention period should be independent of the
> grace period, so I think this is more than a documentation fix (see comments
> below)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
