Thanks. Yes, you're right re (2), I'm new to this stuff. I've currently got code which scans the GlobalKTable in a punctuator and inverts it to another data structure, which I cache (a minute seems suitable for my application), as the only "obvious" way of getting data out of a GlobalKTable (other than the join which is what it's there for in the first place). (This only seems practical because the amount of data I'm handling here is quite small.)
But now the requirements have changed and I'm struggling with handling deletions in the new requirements ... such is life ... Tim Ward -----Original Message----- From: Matthias J. Sax <matth...@confluent.io> Sent: 21 August 2019 07:09 To: users@kafka.apache.org Subject: Re: Restructure data in GlobakKTable Sounds reasonable. Note that a `GlobalKTable` is designed to read data from an topic only without further processing. That's why you don't get any "output stream". For fault-tolerance, the input topic is used, and hence, there is also no "changelog topic". I agree that option (1) is not ideal with regard to separation of concerns / decoupling. For option (2), why do you need the application to re-read the topic? StateStores are by default fully fault-tolerant and hence, if you so a `builder.stream....` and create a StateStore/KTable, on restart the state will still be there. Or maybe I miss understand your question. -Matthias On 8/9/19 6:32 AM, Tim Ward wrote: > I can arrange for there to be a topic which creates a GlobalKTable which > satisfies one of the lookup needs for my Streams application. Basically this > is keyed on "widget ID" and the value includes a "foobar ID". So this lets me > do a join on "widget ID" and find out the "foobar ID", and that works fine. > > **BUT** at some points in my application I also need to do a lookup the other > way around, from "foobar ID" to "widget ID" (yes, this will produce multiple > values, but given the application "multiple" is usually one and is unlikely > ever to be more than a handful). > > If I could get a changelog stream off the GlobalKTable I could use this to > create and maintain a new table which does lookups the other way, but I can't > see any way of getting a stream off the GlobalKTable? > > I could, I suppose, do either of > > (1) Arrange for the source of the input topic to create two topics, with the > data keyed appropriately, and create two GlobalKTables from the two topics. > But this is very clunky, as it requires the other application to have to know > about and supply this application's internal data structure needs. > > (2) Build a separate topology reading from the original topic, restructuring > the data as needed, and building a state store. But how do I ensure that this > topology will re-read the (compacted) input topic from the start every time? > (I'm assuming StreamsBuilder.GlobalTable() knows how to do this, but I don't > know how to do this if using StreamsBuilder.stream().) > > What am I missing please? > > Tim Ward > > This email is from Origami Energy Limited. The contents of this email and any > attachment are confidential to the intended recipient(s). If you are not an > intended recipient: (i) do not use, disclose, distribute, copy or publish > this email or its contents; (ii) please contact Origami Energy Limited > immediately; and then (iii) delete this email. For more information, our > privacy policy is available here: https://origamienergy.com/privacy-policy/. > Origami Energy Limited (company number 8619644) is a company registered in > England with its registered office at Ashcombe Court, Woolsack Way, > Godalming, GU7 1LQ. > This email is from Origami Energy Limited. The contents of this email and any attachment are confidential to the intended recipient(s). If you are not an intended recipient: (i) do not use, disclose, distribute, copy or publish this email or its contents; (ii) please contact Origami Energy Limited immediately; and then (iii) delete this email. For more information, our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy Limited (company number 8619644) is a company registered in England with its registered office at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
