Thanks, based on this we will re-evaluate the use of internal topics. The
main motivation for using the internal changelog topics was to avoid
duplication of data and have an easy way to access the update stream of any
state store.
Best,
Mikael
On Fri, Nov 25, 2016 at 9:52 AM Michael Noll
Mikael,
> Sure, I guess the topic is auto-created the first time I start the
topology
> and the second time its there already. It could be possible to create
> topics up front for us, or even use an admin call from inside the code.
Yes, that (i.e. you are running with auto-topic creation
Sure, I guess the topic is auto-created the first time I start the topology
and the second time its there already. It could be possible to create
topics up front for us, or even use an admin call from inside the code.
That said, as a user, I think it would be great with a function in the
Kafka
Mikeal,
When you use `through(..)` topics are not created by KafkaStreams. You need
to create them yourself before you run the application.
Thanks,
Damian
On Thu, 24 Nov 2016 at 11:27 Mikael Högqvist wrote:
> Yes, the naming is not an issue.
>
> I've tested this with the
Yes, the naming is not an issue.
I've tested this with the topology described earlier. Every time I start
the topology with a call to .through() that references a topic that does
not exist, I get an exception from the UncaughtExceptionHandler:
Uncaught exception
> 1) Create a state store AND the changelog
> topic 2) follow the Kafka Streams naming convention for changelog topics.
> Basically, I want to have a method that does what .through() is supposed to
> do according to the documentation, but without the "topic" parameter.
I understand what you are
Hi Michael,
thanks for the extensive explanation, and yes it definitely helps with my
understanding of through(). :)
You guessed correctly that I'm doing some "shenanings" where I'm trying to
derive the changelog of a state store from the state store name. This works
perfectly fine with with a
> - Also, in case you want to do some shenanigans (like for some tooling
you're building
> around state stores/changelogs/interactive queries) such detecting all
state store changelogs
> by doing the equivalent of `ls *-changelog`, then this will miss
changelogs of KTables that are
> created by
Mikael,
regarding your second question:
> 2) Regarding the use case, the topology looks like this:
>
> .stream(...)
> .aggregate(..., "store-1")
> .mapValues(...)
> .through(..., "store-2")
The last operator above would, without "..." ellipsis, be sth like
`KTable#through("through-topic",
Hi Eno,
1) Great :)
2) Yes, we are using the Interactive Queries to access the state stores. In
addition, we access the changelogs to subscribe to updates. For this reason
we need to know the changelog topic name.
Thanks,
Mikael
On Tue, Nov 22, 2016 at 8:43 PM Eno Thereska
HI Mikael,
1) The JavaDoc looks incorrect, thanks for reporting. Matthias is looking into
fixing it. I agree that it can be confusing to have topic names that are not
what one would expect.
2) If your goal is to query/read from the state stores, you can use Interactive
Queries to do that (you
Sorry for being unclear, i'll try again :)
1) The JavaDoc for through is not correct, it states that a changelog topic
will be created for the state store. That is, if I would call it with
through("topic", "a-store"), I would expect a kafka topic
"my-app-id-a-store-changelog" to be created.
2)
I cannot completely follow what want to achieve.
However, the JavaDoc for through() seems not to be correct to me. Using
through() will not create an extra internal changelog topic with the
described naming schema, because the topic specified in through() can be
used for this (there is no point
Hi Mikael,
If you perform an aggregate and thus create another KTable, that KTable will
have a changelog topic (and a state store that you can query with Interactive
Queris - but this is tangential). It is true that source KTables don't need to
create another topic, since they already have
14 matches
Mail list logo