Re: How do we usually handle Node disconnected issue for kafka producer

, Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Sachin, > > Why do you want to change the default settings? If the connection is open > and unused, > then it is fair to close the connection after the timeout and reopen it > when required. > > On Fri,

How do we usually handle Node disconnected issue for kafka producer

Hi, I am using a Kafka producer java client by vert.x framework. https://vertx.io/docs/apidocs/io/vertx/kafka/client/producer/KafkaProducer.html There is a producer setting in kafka: connections.max.idle.ms = 54 So if there are no records to produce then after 9 minutes I get this in my

Re: best approach to rotate the logs and archive on Kubernetes

I think if you just log to the console, kubernetes will manage the log rotation for you. https://kubernetes.io/docs/concepts/cluster-administration/logging/ You can use "kubectl logs" command to fetch the logs or use some logging agent to move the logs centrally. On Tue, Aug 4, 2020 at 8:15 PM

Re: How to add partitions to an existing kafka topic

consistently route the > > same key data, which is not resolved today. > > > > Boyang > > > > On Wed, Apr 15, 2020 at 7:23 AM Sachin Mittal > wrote: > > > > > Hi, > > > We have a kafka streams application which runs multiple instances and

How to add partitions to an existing kafka topic

Hi, We have a kafka streams application which runs multiple instances and consumes from a source topic. Producers produces keyed messages to this source topic. Keyed messages are events from different sources and each source has a unique key. So what essentially happens is that messages from

Re: Get after put in stateStore returns null

You can try to check the file which rocksdb creates for the state store to see if value is stored for the key. What I suspect is that get may be getting called before put in your topology. Try retrieving the value from state store further downstream. Thanks Sachin On Thu, 2 Apr 2020, 02:35

Re: Are RocksDBWindowStore windows hopping or sliding?

the exact same manner for hopping window > (compare the code I shared in the last reply). Even if windows are > overlapping, the grouping-key+window-start-timestamp is a unique > primary key for each window. > > - -Matthias > > On 2/27/20 9:26 AM, Sachin Mittal wrote: > > Hi,

Re: Kafka JMX monitoring

How about jmxterm. https://cwiki.apache.org/confluence/display/KAFKA/jmxterm+quickstart I have found this very helpful. On Thu, Mar 19, 2020 at 3:51 PM Gioacchino Vino wrote: > Hi, > > > I would like to share my experience with you. > > Even if I found out that prometheus is the a simple and

Re: Passing states stores around

Well I can suggest is that you can access state store via processor's context. Once you have the reference of the state store it can be passed to different classes via the same reference. On Tue, Mar 10, 2020 at 8:57 AM Navneeth Krishnan wrote: > Hi John, > > I'm using PAPI to create my

Re: Are RocksDBWindowStore windows hopping or sliding?

tation for hopping windows > (that we use for tumbling windows, too): > > https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamWindowAggregate.java > > > -Matthias > > On 2/26/20 9:47 PM, Sachin Mittal wrote: > > Hi, &

Re: KStream.groupByKey().aggregate() is always emitting record, even if nothing changed

ng any repartitioning. > > Best, > Bruno > > On Tue, Feb 25, 2020 at 6:43 PM Sachin Mittal wrote: > > > > Hi, > > This is really getting interesting. > > Now if we don't want a record to be emitted downstream only way we can do > > is via transform or (flatTran

Are RocksDBWindowStore windows hopping or sliding?

Hi, So far how I have understood is that when we create a rocksdb window store; we specify a window size and retention period. So windows are created from epoch time based on size, say size if 100 then windows are: [0, 100), [100, 200), [200, 300) ... Windows are retained based on retention

Re: KStream.groupByKey().aggregate() is always emitting record, even if nothing changed

Hi, This is really getting interesting. Now if we don't want a record to be emitted downstream only way we can do is via transform or (flatTransform). Since we are now reverting the fix for null record in transformValues and rather change the docs, doesn't this add bit of confusion for users.

Re: How to set the log level of Kafka clients?

e to know that the log4j > properties file name can be provided in the Java runtime option as - > Dlog4j.configuration=log4propertiesFile. > > Any pointers on whether or how it can be done programmatically would be > greatly helpful. > > Regards, > Pradhan. > > On Sun 23 Fe

Re: How to set the log level of Kafka clients?

I use log4j properties. I think there must be a way to pass log4j settings programmatically as well. If you are using java based producers/consumers then you can set log4j properties and make them available to you classpath before starting those applications. On Sun, Feb 23, 2020 at 9:54 PM

Re: Can we use transform to exchange data between two streams

e existing in-built > operators, because what you've described here really sounds like a > three-way join to me except for streamB it is not joined by key but by time > window. > > Guozhang > > > > On Fri, Feb 21, 2020 at 8:31 PM Sachin Mittal wrote: > > > Hi, >

Re: Understanding concept of grace in JoinWindows

stream application that produces the > data. > > Btw: I gave a talk about time semantics at Kafka Summit recently, so > you might want to check out the recording (there will be a follow up > talk at Kafka Summit London in April focusing on time semantics in > Kafka Streams): > >

Re: Understanding concept of grace in JoinWindows

esult > s2 -> k1 (out-of-order record, within the window: insert into store + join) > -> 2 result2 , > > Because we changes the window size, we get 5 result records instead of 2 > (or 1) as in the first two examples. > > Does this make sense? > > > -Matthias >

There seems some issue with javadoc on KStream#transformValues

Hi, The javadoc of this method states: If the return value of ValueTransformer#transform() is null, no records are emitted. However when I check the underlying processor for the same:

Re: Can we use transform to exchange data between two streams

mBB) results are only > needed for populating the store, you can use `process(..)` instead of > `transform(..)` as it would not need a return value since it would be the > end of this branch of the topology. > > > Guozhang > > On Thu, Feb 20, 2020 at 7:32 PM Sachin Mittal wrote: >

Understanding concept of grace in JoinWindows

Hi, Reading the kafka docs I see that grace period is defined as: the time to admit late-arriving events after the end of the window I however have not understood as when to use it? If I see that some records are arriving after the end of the window and hence not included in the join, should I

Re: Can we use transform to exchange data between two streams

time-range (since you used fetchAll() > which would return ALL the key-values falling into that range) I think this > would be the way to do it. > > > Guozhang > > > On Wed, Feb 19, 2020 at 10:25 PM Sachin Mittal wrote: > > > Hi, > > I have two streams and

Can we use transform to exchange data between two streams

Hi, I have two streams and I want to enrich stream2 records based off stream1 records. I really cannot join those two streams is because there is no common key between them. Hence only way I can do that is using a timestamp field property. This is how I have built my pipeline. . //create and

wanted to understand WindowStore#fetchAll(timeFrom, timeTo)

Hi, Say I want to fetch all the keys for store.fetchAll(8, 12) +---+ | key | start time | end time | +---++--+ | A | 10 |20| +---++--+ | A | 15 |25|

Re: Kafka streams DSL going into infinite loop

ns, ie, is there multiple > sub-topologies that are connected via repartition topics, or is it one > single sub-topology? > > > -Matthias > > On 1/30/20 10:29 AM, Sachin Mittal wrote: > > Hi, > > In a case I have found that when I define my topology using streams DSL &

Kafka streams DSL going into infinite loop

Hi, In a case I have found that when I define my topology using streams DSL it tends to go into infinite loop. This usually happens if I start my stream and shut it down and restart it again after a while (by that time source topic has moved ahead). Stream processing seems to be stuck in a loop

Re: KTable Suppress not working

Hi, As far as my understanding goes, aggregated result for a window is not included in next window. Window would stay in state store till it gets deleted based on certain setting however aggregated result for that window will include only the records that occur within the window duration. If you

Re: Kafka connect not loading classes from uber jar

these warning and what may be causing it. Thanks Sachin On Thu, Jan 16, 2020 at 2:12 PM Sachin Mittal wrote: > Hi, > I am following the documentation under: > https://kafka.apache.org/documentation/#connect_user > https://docs.confluent.io/current/connect/userguide.html > For testing I am u

Kafka connect not loading classes from uber jar

Hi, I am following the documentation under: https://kafka.apache.org/documentation/#connect_user https://docs.confluent.io/current/connect/userguide.html For testing I am using standalone mode. And I am using kafka_2.12-2.3.1. So I have defined: plugin.path=/path/to/plugin/dir In file:

Any available cassandra source connector for kafka

Hi, I was looking for any cassandra source connector to read data from cassandra column family and insert the data into kafka topic. Are you folks aware of any community supported version of such a tool. I found one such: https://docs.lenses.io/connectors/source/cassandra.html However I am not

Re: Streams, Kafka windows

You can try to convert the final resultant stream to table. Check this page for more info: https://docs.confluent.io/current/streams/faq.html#how-can-i-convert-a-kstream-to-a-ktable-without-an-aggregation-step This way table would always contain the latest (single) record for a given key. Sachin

Re: Kafka Streams: out-of-order events when processing multiple topics

I had encountered similar problem and I was using stream to table join. However if a record arrives late in the table, it would not join as per stream to table join semantics. Unfortunately windowing is not supported in stream to table join. I suggest to use stream to stream join with appropriate

Re: Streams Newbie Question - Deployment and Management of Stream Processors

I think literature on confluent/ASF and also the community support here is best to learn about streaming. On Mon, Jan 13, 2020 at 6:47 PM M. Manna wrote: > Hey Sachin, > > On Mon, 13 Jan 2020 at 05:12, Sachin Mittal wrote: > > > Hi, > > The way I have used streams proc

Re: Streams Newbie Question - Deployment and Management of Stream Processors

Hi, The way I have used streams processing in past; use case to process streams is when you have a continuous stream of data which needs to be processed and used by certain applications. Since in kafka streams can be a simple java application, this application can run in its own JVM which is

Re: designing a streaming task for count and event time difference

Try something like this: stream .groupBy( (key, value) -> value.userId ) .aggregate( () -> new Session(), (aggKey, newValue, aggValue) -> { aggValue.userId = newValue.userId if (newValue.start) { aggValue.start = newValue.start aggValue.duration = 0

Kafka trunk vs master branch

Hello Folks, I just wanted to know what commits goes into what branch. I see trunk branch which seems default and latest. I also see master branch which seems bit behind trunk. I also see different versions branches like 2.2, 2.3 and 2.4 which are also actively updated. I wanted to know when

Re: Is there a way to prevent duplicate messages to downstream

inc/kafka-streams-examples/blob/5.3.1-post/src/test/java/io/confluent/examples/streams/EventDeduplicationLambdaIntegrationTest.java > . > > > Also, what is the reason for wanting to eliminate duplicates? Do you have > downstream aggregators that you don't want to incorrectly count dup

Is there a way to prevent duplicate messages to downstream

Hi, I am using streams and I get messages like: (K, V) (A, a), (B, b), (C, c), (A, a), (C, c), (A, a) . I wanted to define a topology which would filter out duplicate messages from upstream. I want to know if this is possible? The code I have written to do this is something like this:

Re: How are custom keys compared during joins

for same key, it won't get joined. So now I have switched back to stream to stream window join. Let me know if there is any other way to handle such a case. Thanks Sachin On Sat, Dec 7, 2019 at 12:08 PM Sachin Mittal wrote: > Hi, > I am facing some weird problems when joining two s

Re: How to set concrete names for state stores and internal topics backed by these

s me irrationally angry, and I'm hoping we can > find a way to fix it, if we ever change the Materialized builder interface. > > Hope this helps, > -John > > On Fri, Dec 6, 2019, at 11:15, Sachin Mittal wrote: > > Hi, > > In my application I have names of internal topics l

Re: What timestamp is used by streams when doing windowed joins

to > non-deterministic programs with unclear semantics. That's why we pretty > much stick to event time in the Streams DSL. > > Finally, yeah, if you just want to process records in the same order they > appear in the topics, then LogAppendTime might be better. > > I hope this h

How are custom keys compared during joins

Hi, I am facing some weird problems when joining two streams or a stream/table. The joined stream does not contain all the joined records. Also note that my keys are custom keys for which I have implemented equals and hashcode method Is there something else also I need to do to ensure key1 ===

What timestamp is used by streams when doing windowed joins

Hi, I have noticed some issues when doing stream to stream windowed joins. Looks like my joined stream does not include all the records. Say I am doing join like this: stream1.join( stream2, (lv, rv) -> ..., JoinWindows.of(Duration.ofMinutes(5)),

Kafka Streams Topology describe format

Hi, I am just posting a section of my topology to basically understand what describe method actually displays. What can we understand just by looking at the topology (like what does --> and <-- arrows represent).

How to set concrete names for state stores and internal topics backed by these

Hi, In my application I have names of internal topics like this: ss-session-application-KSTREAM-JOINOTHER-59-store-changelog-0 ss-session-application-KSTREAM-JOINTHIS-49-store-changelog-0 ss-session-application-KSTREAM-OUTEROTHER-50-store-changelog-0

Re: Case of joining multiple streams/tables

are not fulfilled by the DSL > you can use the Processor API to implement your own join. > > See the following StackOverflow question for more details on joins. > > https://stackoverflow.com/questions/53485632/kafka-streams-implementing-joining-using-the-processor-api > > Bes

Case of joining multiple streams/tables

Hi, I have checked the documentation and what I see that we can join two streams or tables at a given time. I have a case where I have multiple streams which I need to join based on common key. As of now I am first joining two and the result of that with next and so on. Is there a way or any

Re: [ANNOUNCE] Apache Kafka 2.2.2

Does this release has fix for that critical windows bug of not able to delete topics? If not then under which release we can expect the same. Thanks and Regards Sachin On Mon, Dec 2, 2019 at 5:55 PM Karolis Pocius wrote: > 2.2.2 is a bugfix release, it contains some of the fixes from 2.3.0/1,

Re: Deleting topics in windows - fix estimate or workaround

for a contract and demand windows support. > Then magically it happens :) > > On Sun, Dec 1, 2019 at 12:47 PM Sachin Mittal wrote: > > > Hi, > > Firstly development environment is windows and I have been able to run > > almost any big data framework on windows

Re: Deleting topics in windows - fix estimate or workaround

to run it on Windows > > > On Sun, Dec 1, 2019 at 2:22 AM Sachin Mittal wrote: > > > Hi All, > > I hope we are well aware of the critical bug in windows where kafka > crashes > > when we delete a topic. This affects other areas too like in stream > > pr

Deleting topics in windows - fix estimate or workaround

Hi All, I hope we are well aware of the critical bug in windows where kafka crashes when we delete a topic. This affects other areas too like in stream processing when trying to reset a stream using StreamsResetter. Right now only workaround I have found is to stop zookeeper and kafka server and

Re: Kafka Streams with embedded RocksDB

I have checked rocksdbjni-5.0.1.jar included in kafka_2.12-0.11.0.0.tgz and librocksdbjni dll is included. I don't think you have to build anything extra fir windows OS. On Sat, Oct 7, 2017 at 11:39 PM, Valentin Forst wrote: > Hi Ted, > > Thank you for your reply... > Yes,

Re: Kafka Streams with embedded RocksDB

Well as far as I remember this was an old issue with 0.10.0.x or something. In 0.10.2.x librocksdbjni dll is part of rocksdbjni-5.0.1.jar so there is no need to build separately for windows. Did something got changed in 0.11.x ? Thanks Sachin On Fri, Oct 6, 2017 at 10:00 PM, Ted Yu

Kafka streams application failed after a long time due to rocks db errors

Hi, We have been running a clustered kafka streams application and say after 3 months or so of uninterrupted running few threads of couple of instances failed. We checked the logs and we found these two common stack traces pointing to underlying cause of fetch and put operations of rocksdb. Cause

Re: Is there a way in increase number of partitions

ent. > function.Function-org.apache.edgent.function.Function-org. > apache.edgent.function.Function-org.apache.edgent.function.Function- > > > > On 21 August 2017 at 12:02, Sachin Mittal <sjmit...@gmail.com> wrote: > > > Hi, > > I have a topic which has four par

Re: Why does kafka-consumer-groups show the topics written to too in its describe

. Thanks Sachin On Sun, Aug 6, 2017 at 5:57 PM, Sachin Mittal <sjmit...@gmail.com> wrote: > Hi, > I am executing following command > bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server > localhost:9092 --describe --group new-part-advice > > It giv

Is there a way in increase number of partitions

Hi, I have a topic which has four partitions and data is distributed among those based on a specified key. If I want to increase the number of partitions to six how can I do the same and also making sure that messages for a given key always go to one (specific) partition only. Will the existing

Why does kafka-consumer-groups show the topics written to too in its describe

Hi, I am executing following command bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --describe --group new-part-advice It gives output like GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER

Can we use builder.stream twice in single application

Hi, I have a multiple high level streams DSL to execute. In first it reads from a source topic processes the data and sends the data to a sink. In second it again reads from same source topic processes the data and sends it to a different topic. For now these two operations are independent.

Re: Kafka compatibility matrix needed

for an existing topic. That is not supported. > > As usual, you should test this in a non production environment first. Also, > one additional comment inline. > > On Wed, Jul 19, 2017 at 1:19 AM, Sachin Mittal <sjmit...@gmail.com> wrote: > > > > I btw have doubt that different

Re: Kafka compatibility matrix needed

t; > > I personally found that the 0.8.x clients do not work with 0.10.0. We > > upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and > then > > Kafka produce/consume worked fine. > > > > --John > > > > On Tue, Jul 18, 2017 at 6:36 AM, Sa

Re: Kafka compatibility matrix needed

com> wrote: > All broker versions support all older client versions > > On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal <sjmit...@gmail.com> > wrote: > > > Hi, > > This gives me some information but still not the complete picture. > > > > It says:

Re: Kafka compatibility matrix needed

s in the > Kafka cluster before upgrading your clients. Version 0.11.0 brokers support > 0.8.x and newer clients. > > Hope that helps. > > Cheers, > > Michał > > On 18/07/17 08:17, Sachin Mittal wrote: > > Hi, > I would like some help/information on what clie

Kafka compatibility matrix needed

Hi, I would like some help/information on what client versions are compatible with what broker versions in kafka. Some table like this would be good server client 0.80.9 0.10 0.11 0.8 yes ? ?? 0.9 ?yes ??

How can we re-key and re-partition records based on that key

Hi, I have a topic of records (K1, V1). This topic is partitioned based on the K1. I use multiple streams instances to process these records based on that key so that each instance process certain partitions only. Now I want to re-key the records in that topic with a format (K2, V2) and push

Do we have to query localWindowStore in same java instance we are creating the store

Hi, I have created a simple window store to count occurrences of a given key. My pipeline is: TimeWindows windows = TimeWindows.of(n).advanceBy(n).until(30n); final StateStoreSupplier supplier = Stores.create("key-table") .withKeys(Serdes.String())

Re: What could cause unavailable partitions issue and how to fix it?

Folks any input on this? Would help us a lot in case we get into this issue in future as right now only option is to delete the changelog table. On Sat, Jun 10, 2017 at 12:19 AM, Sachin Mittal <sjmit...@gmail.com> wrote: > In course of our streaming application we discovered tha

Re: getting intermittent TimeoutException at producer side in streams application

of streams topics (e.g., > changelogs and repartition topics) to 2 or 3 using > StreamsConfig.REPLICATION_FACTOR_CONFIG. > > Thanks > Eno > > > > On 9 Jun 2017, at 20:01, Sachin Mittal <sjmit...@gmail.com> wrote: > > > > Hi All, > &g

Re: getting intermittent TimeoutException at producer side in streams application

> This would usually indicate that may indicate that there is a connectivity > issue with the brokers. You would need to correlate the logs etc on the > brokers with the streams logs to try and understand what is happening. > > Thanks, > Damian > > On Sun, 18 Dec 2016 at 07:

What could cause unavailable partitions issue and how to fix it?

In course of our streaming application we discovered that it went into hung state and upon further inspection we found that some of the partitions had no leaders assigned. Here is the description of topic: # bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic

Re: Kafka Streams Failed to rebalance error

One instance with 10 threads may cause rocksdb issues. What is the RAM you have? Also check CPU wait time. Many rocks db instances on one machine (depends upon number of partitions) may cause lot of disk i/o causing wait times to increase and hence slowing down the message processing causing

Re: weird SerializationException when consumer is fetching and parsing record in streams application

bove ("Error deserializing key/value for partition..."), and the > >> Fetcher is swallowing the more specific SerializationException of > >> `String.Serdes()` (but it will include the original exception/Throwab

Re: Large Kafka Streams deployment takes a long time to bootstrap

with 1 > > > > core as described in https://issues.apache.org/ > jira/browse/KAFKA-5174 > > < > > > > https://issues.apache.org/jira/browse/KAFKA-5174>. > > > > > > > > If you have 1 core, the workaround is to change a config as des

Re: Large Kafka Streams deployment takes a long time to bootstrap

Note on few things. Set changelog topic delete retention time to as less as possible if the previous values for same key are not needed and can be safely cleaned up. Set segment size and segment retention time also low so older segments can be compacted and cleaned up. Set delete ratio to be

Re: Failure on timestamp extraction for kafka streams 0.10.2.0

s.apache.org/jira/browse/KAFKA-4861>. > > Thanks > Eno > > On May 2, 2017, at 8:43 AM, Sachin Mittal <sjmit...@gmail.com> wrote: > > > > The timestamp of the message is out of acceptable range > >

Failure on timestamp extraction for kafka streams 0.10.2.0

Hi, We are getting this exception right when we start the stream. org.apache.kafka.common.KafkaException: Unexpected error from SyncGroup: The timestamp of the message is out of acceptable range. at

Re: Kafka streams internal topic data retention

http://kafka.apache.org/documentation.html#topic-config Check this. You can use *--alter * To override/add the default config. retention.ms can be used to set topic level config. For internal topics I suppose you need to provide a topic config map before creating internal topics. Example:

Re: Stream applications dying on broker ISR change

I had a question about this setting ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, Integer.toString(Integer.MAX_ VALUE) How would the broker know if a thread has died or say we simply stopped an instance and needs to be booted out of the group. Thanks Sachin On Mon, Apr 24, 2017 at 5:55 PM, Eno

Re: possible kafka bug, maybe in console producer/consumer utils

Just to point to you all, I also get similar exception in my streams application when producer is trying to commit something to changelog topic. Error sending record to topic test-stream-key-table-changelog org.apache.kafka.common.errors.TimeoutException: Batch containing 2 record(s) expired due

Re: Streams error handling

We are also catching the exception in serde and returning null and then filtering out null values downstream so as they are not included. Thanks Sachin On Thu, Apr 13, 2017 at 9:13 PM, Mike Gould wrote: > Great to know I've not gone off in the wrong direction > Thanks > >

Re: How to get metrics from Kafka?

check this http://kafka.apache.org/documentation.html#monitoring You can use any jmx client to hook up to the running server jvm. https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters or use jconsole. I use https://cwiki.apache.org/confluence/display/KAFKA/jmxterm+quickstart and works

Re: auto.offset.reset for Kafka streams 0.10.2.0

As far as I know default is latest, if no offsets are found. Otherwise it starts from the offset. On Tue, Apr 11, 2017 at 8:51 AM, Mahendra Kariya wrote: > Hey All, > > Is the auto offset reset set to "earliest" by default in Kafka streams > 0.10.2.0? I thought

Re: In kafka streams consumer seems to hang while retrieving the offsets

ON_ID_CONFIG, ID); > ... > props.put(ProducerConfig.RETRIES_CONFIG, 10); > props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, > Integer.toString(Integer.MAX_VALUE)); > > This doesn't address the RocksDB issue though, still looking into that. > > Thanks > Eno > > > On 9 Apr 2017

Re: In kafka streams consumer seems to hang while retrieving the offsets

m those > nodes? Also would be useful to know the config (e.g., how many partitions > do you have and how many app instances.) > > Thanks > Eno > > > On 9 Apr 2017, at 04:45, Sachin Mittal <sjmit...@gmail.com> wrote: > > > > Hi, > > In my streams appli

In kafka streams consumer seems to hang while retrieving the offsets

Hi, In my streams applications cluster in one or more instances I see some threads always waiting with the following stack. Every time I check on jstack I see the following trace. Is this some kind of new deadlock that we have failed to identify. Thanks Sachin here is the stack trace:

Re: Topic deletion

Do you have delete.topic.enable=true uncommented or present in server.properties On Thu, Apr 6, 2017 at 11:03 PM, Adrian McCague wrote: > Hi all, > > I am trying to understand topic deletion in kafka, there appears to be > very little documentation or answers on how

Re: compaction not happening for change log topic

t; -Matthias > > On 3/31/17 11:49 AM, Sachin Mittal wrote: > > Hi, > > I have noticed that many times change log topics don't get compacted. The > > segment log file is always 1 GB. > > > > So I would like to know how and when compaction comes into play. > >

Re: Difference between with and without state store cleanup streams startup

he/kafka/blob/trunk/streams/src/ > main/java/org/apache/kafka/streams/processor/internals/ > StoreChangelogReader.java#L99 > > called here: > https://github.com/apache/kafka/blob/trunk/streams/src/ > main/java/org/apache/kafka/streams/processor/internals/ > StreamThread.java#L1294 >

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

you can increase ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG -Matthias On 3/31/17 11:32 AM, Sachin Mittal wrote: > Hi, > So I have added the config ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE > and the NotLeaderForPartitionException is gone. > > However we see a new except

compaction not happening for change log topic

Hi, I have noticed that many times change log topics don't get compacted. The segment log file is always 1 GB. So I would like to know how and when compaction comes into play. is there a way we can get the topic compacted say trigger compaction after x seconds of a given message or a given

Difference between with and without state store cleanup streams startup

Hi, There are two ways to re start a streams application 1. executing streams.cleanUp() before streams.start() This cleans up the local state store. 2. Just by calling streams.start() What are the differences between two. As I understand in first case it will try to create local state store by

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

. Is there any other setting that we should try to avoid such errors which causes stream thread to die. Thanks Sachin On Sun, Mar 26, 2017 at 1:39 AM, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Sachin, > > Not in this case. > > Thanks > Eno > > > On Mar 25, 2017,

Re: weird SerializationException when consumer is fetching and parsing record in streams application

built-in `Serdes.String()` > > does > > >> not try-catch deserialization errors, and from a quick look at the > > source > > >> it seems that the `Fetcher` class (clients/src/main/java/org/apa > > >> che/kafka/clients/consumer/internals/Fetcher.java) is t

Re: weird SerializationException when consumer is fetching and parsing record in streams application

bove ("Error deserializing key/value for partition..."), and the > >> Fetcher is swallowing the more specific SerializationException of > >> `String.Serdes()` (but it will include the original exception/Throwable > in > >> its own SerializationException). > >

Re: weird SerializationException when consumer is fetching and parsing record in streams application

o read the message. If you did this in debug mode you might find > out some more information. > > > On Thu, 30 Mar 2017 at 11:50 Sachin Mittal <sjmit...@gmail.com> wrote: > > > Well I try to read that offset via kafka-console-consumer.sh too and it > > fails with same

Re: weird SerializationException when consumer is fetching and parsing record in streams application

g what the message is? Might be easier to debug? Like > you say, it is failing in the consumer. > Thanks, > Damian > > On Thu, 30 Mar 2017 at 10:35 Sachin Mittal <sjmit...@gmail.com> wrote: > > > I think I am following the third option. > > > > My pipelin

Re: weird SerializationException when consumer is fetching and parsing record in streams application

dress such > poison pill problems. If you have feedback on that front, feel free to > share it with us. :-) > > -Michael > > > > > On Wed, Mar 29, 2017 at 10:07 PM, Sachin Mittal <sjmit...@gmail.com> > wrote: > > > Hi, > > This is for first time

weird SerializationException when consumer is fetching and parsing record in streams application

Hi, This is for first time we are getting a weird exception. After this the streams caches. Only work around is to manually seek and commit offset to a greater number and we are needing this manual intervention again and again. Any idea what is causing it and how can we circumvent this. Note

Re: 3 machines 12 threads all fail within 2 hours of starting the streams application

e helpful to understand, what scenario works and what not. > Right now you go from > single-threaded-sinlge-instance-with-no-partitioning to > multi-threaded-multiple-instances-and-partitioned -- that's a big step > to reason about the situation. > > > -Matthias > > > On

Re: kafka streams 0.10.2 failing while restoring offsets

t; long do we wait before we give up? > > I think in this case the exception is the right thing to do > > On Mon, 27 Mar 2017 at 09:24 Sachin Mittal <sjmit...@gmail.com> wrote: > > Hi, > These are the logs > https://www.dropbox.com/s/2t4ysfdqbtmcusq/complete_84_85

  1   2   3   >