Jan, Sure I would love to hear what you did for non-key joins. Last time we chatted there are discussions on the ordering issue, that we HAVE TO augment the join result stream keys as a combo of both, which may not be elegant as used in the DSL.
For your proposed solution, it seems you did not do that on the DSL but at the PAPI layer, right? Guozhang On Tue, Feb 21, 2017 at 6:05 AM, Jan Filipiak <jan.filip...@trivago.com> wrote: > Just a little note here: > > if you can take all rows of the "children" table for each key into memory, > you get get away by using group_by and make a list of them. With this > aggregation the join is straight forward and you can use a lateral view > later to get to the same result. For this you could use the current DSL to > a greater extend. > > Best Jan > > On 21.02.2017 13:10, Frank Lyaruu wrote: > >> I've read that JIRA (although I don't understand every single thing), and >> I >> got the feeling it is not exactly the same problem. >> I am aware of the Global Tables, and I've tried that first, but I seem >> unable to do what I need to do. >> >> I'm replicating a relational database, and on a one-to-many relationship >> I'd like to publish a joined message if either of the source streams >> receives an update. >> >> In the Global Table Wiki: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-99%3A+ >> Add+Global+Tables+to+Kafka+Streams >> >> I see this: >> "The GlobalKTable will only be used for doing lookups. That is, data >> arriving in the GlobalKTable will not trigger the join. " >> >> So how would I go about doing this? >> regards, Frank >> >> >> >> On Tue, Feb 21, 2017 at 10:38 AM, Eno Thereska <eno.there...@gmail.com> >> wrote: >> >> Hi Frank, >>> >>> As far as I know the design in that wiki has been superceded by the >>> Global >>> KTables design which is now coming in 0.10.2. Hence, the JIRAs that are >>> mentioned there (like KAFKA-3705). There are some extensive comments in >>> https://issues.apache.org/jira/browse/KAFKA-3705 < >>> https://issues.apache.org/jira/browse/KAFKA-3705> illustrating why this >>> design is particularly challenging and why Global KTables was chosen >>> instead. I'm not sure if you still want to pursue that original design, >>> since it is not proven to work. >>> >>> Guozhang, perhaps we need to add a note saying that Global KTables is the >>> new design? >>> >>> Thanks >>> Eno >>> >>> On 21 Feb 2017, at 07:35, Frank Lyaruu <flya...@gmail.com> wrote: >>>> >>>> Hi all, >>>> >>>> I'm trying to implement joining two Kafka tables using a 'remote' key, >>>> basically as described here: >>>> >>>> https://cwiki.apache.org/confluence/display/KAFKA/ >>>> >>> Discussion%3A+Non-key+KTable-KTable+Joins >>> >>>> Under the "Implementation Details" there is one line I don't know how to >>>> do: >>>> >>>> >>>> 1. First of all, we will repartition this KTable's stream, by key >>>> computed from the *mapper(K, V) → K1*, so that it is co-partitioned >>>> by >>>> the same key. The co-partition topic is partitioned on the new key, >>>> >>> but the >>> >>>> message key and value are unchanged, and log compaction is turned >>>> off. >>>> >>>> >>>> How do I do that? I've been unable to find any documentation, I've >>>> looked >>>> at the StreamPartitionAssignor, that seems relevant, but I could use >>>> some >>>> help. Does anyone have an example? >>>> >>>> regards, Frank >>>> >>> >>> > -- -- Guozhang
