Matthias, thank you for getting back on this. - I'll just let the
application run for now. We are trying to build a CDC system
leveraging Kafka in order to feed aggregated data into an indexing
solution like Elasticsearch or Opensearch. Maybe after all the Kafka
Streams application will be
Using the DSL, this sounds about right.
I am not worried about the complexity -- KS can handle it, and it's not
uncommon to end up with such topologies.
You might be able to cut down on complexity by not using the DSL, but
the Processor API. It gives you more control, and thus you might be
Sorry for being late with the response - I've been quite busy working
on our Streams application lately.
That leads me back to my initial question. The Folder class contains
multiple fields with FK pointing to the Person table, all of them with
different semantics (customer, billing address,
I see. You need to ensure that you get _all_ Person.
For this case, I guess you are right. You would need to first aggregate
the folder per person:
KTable allPersonFolders =
folder.groupBy((...) -> (folder.customerId, ...))
.aggregate(...)
And in a second step, do a left join:
Thanks so much for taking a look. An FK-table-table join is an inner
join which implies there would be no Person entites without associated
Folders. Unfortunately, that's not the case. That lead me to an
attempt of re-keying the Folder topic by each of the three possible
foreign keys in order to
Thanks for the details. This does make sense.
So it seems you can read all topic as table (ie, builder.table("topic")
-- no need to so `builder.stream().toTable()`).
And you can use the built-in FK-table-table join, and aggregate the result:
KTable result =
folderTable
Matthias, thanks for getting back on this. I'll try to illustrate my
intent with an example as I'm not yet fully familiar with Kafka
(Streams) and its idioms...
Assume classes Person and Folder:
class Person {
Long id;
String firstname;
String lastname;
// some content
}
class Folder {
Both fk1 and fk2 point to the PK of another entity (not shown for
brevity, of no relevance to the question).
It this two independent FK, or one two-column FK?
Ingesting the topic into a Kafka Streams application, how can I re-key
the resulting KTable by both fk1 and fk2?
If you read the
Hi all,
just stumbled upon another Kafka Streams issue that keeps me busy these days.
Assume a (simplified) class A like this:
class A {
private Long id;
private String someContent;
private Long fk1;
private Long fk2;
// Getters and setters accordingly
}
Both fk1 and fk2
