Hi,
Are there any standard benchmarking results available for Kafka Streams ? I am
specifically looking for stateful operations and the cost of
serialization/deserialization which could be a limiting factor if the state per
key can be large and there are frequent updates. What is the typical
doing redundant work and then drop the results on the floor since nothing
was reflected outside.
On the other hand, if the tasks are at least writing to some sink topics,
then zombies would still be detected.
Guozhang
On Thu, Apr 22, 2021 at 10:47 AM Parthasara
Or create a ticket for tracking.
-Matthias
On 4/28/21 12:49 PM, Parthasarathy, Mohan wrote:
> Andrew,
>
> I am not sure I understand. We have built several analytics applications.
We typically use custom aggregations as they are not available direc
Andrew,
I am not sure I understand. We have built several analytics applications. We
typically use custom aggregations as they are not available directly in the
library.
-mohan
On 4/28/21, 12:12 PM, "Andrew Otto" wrote:
I'd assume this is because Kafka Streams is positioned for
more built-in aggregations next to
count(). It did not come up in the community so far, so there was no
investment yet.
-Matthias
On 4/28/21 10:30 AM, Parthasarathy, Mohan wrote:
> Hi,
>
> Whenever the discussion about what streaming framework to use
Hi,
Whenever the discussion about what streaming framework to use for near-realtime
analytics, there is normally a discussion about Spark vs Kafka streaming. One
of the points in favor of Spark streaming is the simple aggregations that are
built-in. See here:
Guozhang,
What does this mean if the changelog topic was disabled ? If thread 2 and
thread 4 are running in two different nodes and a rebalance occurs, thread 2
will not realize it is a zombie without the write to the changelog topic, right
? I am trying to understand the cases under which the
> group and not trigger unnecessary rebalances.
>
> Guozhang
>
> On Thu, Mar 11, 2021 at 6:41 PM Parthasarathy, Mohan
> wrote:
>
> > Hi,
> >
> > I have a use case where messages come in with some key gets assigned
some
>
Hi,
I have a use case where messages come in with some key gets assigned some
partition and the state gets created. Later, key changes (but still contains
the old key in the message) and gets sent to a different partition. I want to
be able to grab the old state using the old key before
op of app1 directly leveraging on its
IQ interface. For some examples code you can refer to
https://github.com/confluentinc/kafka-streams-examples/tree/6.0.0-post/src/main/java/io/confluent/examples/streams/interactivequeries
Guozhang
On Thu, Oct 1, 2020 at 10:40 AM Parthasara
tate stores of app2 through the
interactive query interface, right? This is actually a pretty common use
case pattern for IQ :)
Guozhang
On Wed, Sep 30, 2020 at 1:22 PM Parthasarathy, Mohan
wrote:
> Hi,
>
> A traditional kafka streams applicati
Hi,
A traditional kafka streams application (App1) reading data from a kafka
topic, doing aggregations resulting in some local state. The output of this
application is consumed by a different application(App2) for doing a different
task. Under some conditions, there is an external trigger
nd you can apply is to increase the grace period.
Increasing `max.task.idle.ms` config may also help.
-Matthias
On 9/24/19 12:58 PM, Parthasarathy, Mohan wrote:
> Hi,
>
> Here is a simple example:
>
> Application is reading a topic
Hi,
Here is a simple example:
Application is reading a topic where messages are being received from various
clients identified by “client_id”. The messages are grouped by “client_id” and
windowedBy 10 seconds with grace period of 5 seconds. The event time for the
stream progresses when we
reams side. It's just not implemented yet.
Does this all seem about right to you?
-john
On Wed, Jun 26, 2019 at 12:57 PM Parthasarathy, Mohan
wrote:
>
> Initially it started in the testing. QA reported problems where "events"
were n
repartition.
Can we take a step back and discuss the motivation for forcing the
records to flush out? Is this for testing your app, or is it to drive
some production logic?
Thanks,
-John
On Mon, Jun 24, 2019 at 7:26 PM Parthasarathy, Mohan
wro
t, as you can add another instance to your cluster at any time,
and Streams will scale up, and you'll know that the program is
executing exactly the same way the whole time.
-John
On Sat, Jun 22, 2019 at 4:37 PM Parthasarathy, Mohan
wrote:
>
> I can see the i
o, for most tests, I just
configure the topics to have one partition.
-John
On Fri, Jun 21, 2019 at 3:56 PM Parthasarathy, Mohan
wrote:
>
> That change "In the same partition" must explain what we are seeing.
Unless you see one message per pa
ohn
On Thu, Jun 20, 2019 at 11:42 AM Parthasarathy, Mohan
wrote:
>
> Could you tell me a little more about the delays about the record caches
and how I can disable it ?
>
> If I could summarize my problem:
>
> -A new record with a new
Thanks,
-John
On Wed, Jun 19, 2019 at 3:01 PM Parthasarathy, Mohan
wrote:
>
> We do explicitly set the grace period to zero. I am going to try the new
version
>
> -mohan
>
>
> On 6/19/19, 12:50 PM, "Parthasarathy, Mo
We do explicitly set the grace period to zero. I am going to try the new version
-mohan
On 6/19/19, 12:50 PM, "Parthasarathy, Mohan" wrote:
Thanks. We will give it a shot.
On 6/19/19, 12:42 PM, "Bruno Cadonna" wrote:
Hi Mohan,
that version.
>
> Best,
> Bruno
>
> On Wed, Jun 19, 2019 at 8:37 PM Parthasarathy, Mohan
wrote:
> >
> > No, I have not set any grace period. Is that mandatory ? Have you seen
problems with suppress and windows expiring ?
> >
&
ue, Jun 18, 2019 at 2:04 AM Parthasarathy, Mohan
wrote:
>
> On further debugging, what we are seeing is that windows are expiring
rather randomly as new messages are being processed. . We tested with new key
for every new message. We waited for the window time before replaying new
mes
there an order in the way windows
are closed ? As event time progresses by the new messages arriving, the older
ones should expire. Is that right understanding or not ?
Thanks
Mohan
On 6/17/19, 3:43 PM, "Parthasarathy, Mohan" wrote:
Hi,
We are using suppress in the application. W
Hi,
We are using suppress in the application. We see some state being created at
some point in time. Now there is no new data for a day or two. We send new data
but the old window of data (where we see the state being created) is not
closing i.e not seeing it go through suppress and on to the
Hi,
Is there a way to know about the currently open windows that is currently not
expired yet ? We are trying to track down a problem where some messages are
read by the application but then we don’t see the actual processing of the
message by the application.
Thanks
Mohan
ompaction affects the rebuilding time ? Sorry, I am not
sure I understand what you meant by this in the current context.
-mohan
On 6/10/19, 10:22 AM, "Parthasarathy, Mohan" wrote:
Thanks. That helps me understand why recreating state might take time.
-mohan
long recovery times.
-Matthias
On 6/9/19 2:59 PM, Parthasarathy, Mohan wrote:
> Matt,
>
> Thanks for your response. I agree with you that there is no easy way to
answer this. I was trying to see what others experience is which could simply
Pavel,
Thanks for the pointer. I will take a look.
-mohan
On 6/8/19, 4:29 PM, "Pavel Sapozhnikov" wrote:
I suggest take a look at Strimzi project https://strimzi.io/
Kafka operator deployed in Kubernetes environment.
On Sat, Jun 8, 2019, 6:09 PM Parthasara
ake a look at Strimzi project https://strimzi.io/
>
> Kafka operator deployed in Kubernetes environment.
>
> On Sat, Jun 8, 2019, 6:09 PM Parthasarathy, Mohan
wrote:
>
>> Hi,
>>
>> I have read several articles about this topic. We
Hi,
I have read several articles about this topic. We are soon going to deploy our
streaming apps inside k8s. My understanding from reading these articles is that
stateful set in k8s is not mandatory as the application can rebuild its state
if the state store is not present. Can people share
I am a little confused by what you say. I can see how it has to build the state
when it is not available on restart but i don’t think it will process old
messages from input topics. It should start from the last committed offset
whatever that is before the crash. Could you confirm ? I thought
and you think you should not see?
It is hard to help you without any actual logs.
Best,
Bruno
On Wed, Jun 5, 2019 at 6:52 AM Parthasarathy, Mohan wrote:
>
> Hi,
>
> As mentioned here
>
> https://issues.apache.org/jira/browse/KAFKA-7510
>
> I do see these logs turned on by
Hi,
As mentioned here
https://issues.apache.org/jira/browse/KAFKA-7510
I do see these logs turned on by default. We are running 2.2 and I still see
the error. It looks like the fix is in 2.0. What am I missing ? It seems to be
logged in “WARN” level instead of “debug” level. It spews a huge
Hi,
When applications are restarted, I see it go into rebalancing state. Is it
trying to recreate state from the state store files ? I am seeing an
application stuck for a long time and wondering how to debug the application in
this state ?
Thanks
Mohan
-Matthias
On 5/15/19 10:55 PM, Parthasarathy, Mohan wrote:
> Hi,
>
> I am seeing multiple state store files (.sst) created under each
directory. Is this normal ? If an application crashes
and comes back, would it cause this ?
>
> Thanks
> Mohan
>
Hi,
I am seeing multiple state store files (.sst) created under each
directory. Is this normal ? If an application crashes
and comes back, would it cause this ?
Thanks
Mohan
37 matches
Mail list logo
