Hi Matthias,
I have configured the GlobalKTable to stream from a topic and application
is working fine, however during automated build test cases, sometimes I get
an exception: I believe this could be because of race between actual topic
creation and the service startup (since topic creation may n
Matthias,
I realized that the exception and actual problem is totally different. The
problem was the client was not set with SSL truststore while server is
SSLenabled.
I also found this open bug on kafka
https://issues.apache.org/jira/browse/KAFKA-4493
After setting the SSL properties on stream, I
Thanks... i will try increasing the memory in case you don't spot anything
wrong with the code. Other service also have streams and global k table but
they use spring-kafka, but i think that should not matter, and it should
work with normal kafka-streams code unless i am missing some
configuration/
There is no hook. Only a restore listener, but this one is only used
during startup when the global store is loaded. It's not sure during
regular processing.
Depending on your usage, maybe you can switch to a global store instead
of GlobalKTable? That way, you can implement a custom `Processor` an
Matthias,
I tried with default store as well but getting same error, can you please
check if I am initializing the global store in the right way:
public void setupGlobalCacheTables(String theKafkaServers) {
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
Hi Matthias,
By the way, I used the in-memory global store and the service is giving out
of memory error during startup. Unfortunately i don't have a stack trace
now but when i got stack the first time, the error was coming somewhere
from memorypool.allocate or similar kind of method. If i get the
Ok... got it... is there any hook that I can attach to the global k table
or global store? What I mean here is I want to know when the global store
is updated with data from topic in that case the hook that I specified
should be invoked so i can do some activity like logging that, this will
allow m
For example it could be some "static" information, like a mapping from
zip code to city name.
Something that does usually not change over time.
-Matthias
On 5/25/20 9:55 PM, Pushkar Deole wrote:
> Matthias,
>
> I am wondering what you mean by "Global store hold "axially" data that is
> provide
Matthias,
I am wondering what you mean by "Global store hold "axially" data that is
provided from "outside" of the
app"
will you be able to give some example use case here as to what you mean by
axially data provided from outside app?
On Sat, May 2, 2020 at 1:58 AM Matthias J. Sax wrote:
> Bot
Hi Pushkar,
I’m glad you’ve been able to work through the issues.
The GlobalKTable does store the data in memory (or on disk, depending how you
configure it). I think the in-memory version uses a TreeMap, which is
logarithmic time access. I think you’ll find it sufficiently fast regardless.
T
John,
I think I can get the cache structure modified to make use of GlobalKTable
here so the data can be shared across. I could get information that the
admin data will be uploaded well in advance before main events so the issue
with 'missed joins' won't exists since by the time main events start
Hello John, Matthias
Sorry for bothering you, however this is now getting crazier. Initially I
was under the impression that the cache being hold by application is in the
form of key/value where key is the instance of agentId (e.g. 10) and value
will hold other attributes (and their respective val
Hi Pushkar,
To answer your question about tuning the global store latency, I think the
biggest impact thing you can do is to configure the consumer that loads the
data for global stores. You can pass configs specifically to the global
consumer with the prefix: “ global.consumer.”
Regarding the
Thanks John... appreciate your inputs and suggestions. I have been assigned
recently to this task (of persisting the cache) and haven't been involved
in original design and architecture and agree with all the issues you have
highlighted.
However, at this point, i don't think the application can be
Thanks John... what parameters would affect the latency in case
GlobalKTable will be used and is there any configurations that could be
tuned to minimize the latency of sync with input topic?
On Mon, May 4, 2020 at 10:20 PM John Roesler wrote:
> Hello Pushkar,
>
> Yes, that’s correct. The operat
Hello Pushkar,
Yes, that’s correct. The operation you describe is currently not supported. If
you want to keep the structure you described in place, I’d suggest using an
external database for the admin objects. I’ll give another idea below.
With your current architecture, I’m a little concern
If i understand correctly, Kafka is not designed to provide replicated
caching mechanism wherein the updates to cache will be synchronous across
multiple cache instances.
On Sun, May 3, 2020 at 10:49 PM Pushkar Deole wrote:
> Thanks John.
>
> Actually, this is a normal consumer-producer applicat
Thanks John.
Actually, this is a normal consumer-producer application wherein there are
2 consumers (admin consumer and main consumer) consuming messages from 2
different topics.
One of the consumers consumes messages from a admin topic and populates
data in a cache e.g. lets say agent with agent
Hi Pushkar,
I’ve been wondering if we should add writable tables to the Streams api. Can
you explain more about your use case and how it would integrate with your
application?
Incidentally, this would also help us provide more concrete advice.
Thanks!
John
On Fri, May 1, 2020, at 15:28, Matt
Both stores sever a different purpose.
Regular stores allow you to store state the application computes.
Writing into the changelog is a fault-tolerance mechanism.
Global store hold "axially" data that is provided from "outside" of the
app. There is no changelog topic, but only the input topic (t
thanks... will try with GlobalKTable.
As a side question, I didn't really understand the significance of global
state store which kind of works in a reverse way to local state store i.e.
local state store is updated and then saved to changelog topic whereas in
case of global state store the topic i
Yes.
A `GlobalKTable` uses a global store internally.
You can also use `StreamsBuilder.addGlobalStore()` or
`Topology.addGlobalStore()` to add a global store "manually".
-Matthias
On 4/30/20 7:42 AM, Pushkar Deole wrote:
> Thanks Matthias.
> Can you elaborate on the replicated caching layer p
Thanks Matthias.
Can you elaborate on the replicated caching layer part?
When you say global stores, do you mean GlobalKTable created from a topic
e.g. using StreamsBuilder.globalTable(String topic) method ?
On Thu, Apr 30, 2020 at 12:44 PM Matthias J. Sax wrote:
> It's not possible to modify st
It's not possible to modify state store from "outside".
If you want to build a "replicated caching layer", you could use global
stores and write into the corresponding topics to update all stores. Of
course, those updates would be async.
-Matthias
On 4/29/20 10:52 PM, Pushkar Deole wrote:
> Hi
Hi All,
I am wondering if this is possible: i have been asked to use state stores
as a general replicated cache among multiple instances of service instances
however the state store is created through streambuilder but is not
actually modified through stream processor topology however it is to be
25 matches
Mail list logo