Re: committing offset metadata in kafka streams

om multiple topics and join/merge them, a task gets multiple > partitions (from different topics) assigned. > > > -Matthias > > On 3/9/18 2:42 PM, Stas Chizhov wrote: > >> Also note, that the processing order might slightly differ if you > > process the same data twic

Re: committing offset metadata in kafka streams

e "robust" against different > processing orders (ie, if there are multiple input partitions, you might > get data first for partition 0 and there for partition 1 or the other > way round -- the order per partitions is guaranteed to be in offset order). > > > -Mat

isolated connector deployment

Hi, We do have mesos based infrastructure to run our java-based microservices and we'd like to use it for deploying connectors as well (with benefits of reusing deployment specific knowledge we already have, isolating the load and in general pretty much the same reasons Kafka Streams was designed

Re: committing offset metadata in kafka streams

t time anybody asks for this feature. If > there is a JIRA, maybe somebody picks it up :) > > > -Matthias > > On 3/3/18 6:51 AM, Stas Chizhov wrote: > > Hi, > > > > There seems to be no way to commit custom metadata along with offsets > from > > within Kafka St

committing offset metadata in kafka streams

Hi, There seems to be no way to commit custom metadata along with offsets from within Kafka Streams. Are there any plans to expose this functionality or have I missed something? Best regards, Stanislav.

Re: why number of insync replicats is less than min.insync.replicas?

Hi, it looks like https://issues.apache.org/jira/browse/KAFKA-5970. Try restarting broker 1. Best regards, Stanislav. 2017-11-10 14:00 GMT+01:00 Vitaliy Semochkin : > Hi, > > I have a cluster with 3 brokers (0.11) > when I create a topic with min.insync.replicas=2 and

Mirroring avro topics with copying schemas between registries

Hi, I would like to mirror a topic with avro messages from a production to staging cluster, each cluster having its own schema registry. I would like messages being copied to use schemas from target cluster registry and those schemas copied from source registry if needed. I guess I could do it

Re: Figuring out lag within Java consumer application

Hi, You can get lag as a metric here: https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#metrics() . BR, Stas. 2017-10-13 2:08 GMT+02:00 Stephen Powis : > So I have the same use case as the original poster and had the same issue >

Re: Serve interactive queries from standby replicas

drop on the floor? > > > Guozhang > > On Fri, Oct 6, 2017 at 1:10 PM, Stas Chizhov <schiz...@gmail.com> wrote: > > > Thank you! > > > > I guess eventually consistent reads might be a reasonable trade off if > you > > can get ability to serve read

Re: Serve interactive queries from standby replicas

gt; > >> By the way standby replicas are just extra consumers/processors of input > >> topics? Or is there some custom protocol for sinking the state? > > We use a second consumer, that reads the changlog topic (that is written > by the active store) to update the hot stan

Re: kafka broker loosing offsets?

owest id (e.g. 100 from [100,101,102]). > When crashing other brokers - there is nothing special happening, lag > growing little bit but nothing crazy (e.g. thousands, not millions). > > Is it sounds suspicious? > > On Fri, Oct 6, 2017 at 9:23 PM, Stas Chizhov <schiz...@gmail.com>

Re: Serve interactive queries from standby replicas

might add this functionality at some point though -- but there are no > concrete plans atm. Contributions are always welcome of course :) > > > -Matthias > > On 10/6/17 4:18 AM, Stas Chizhov wrote: > > Hi > > > > Is there a way to serve read read requests from st

Re: kafka broker loosing offsets?

ka, it commits offsets "manually" for us after > event > > handler completed. So it's kind of automatic once there is constant > stream > > of events (no idle time, which is true for us). Though it's not what pure > > kafka-client calls "automatic" (flu

Re: kafka broker loosing offsets?

bably something wrong with broker setup then? > > On Fri, Oct 6, 2017 at 6:58 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > Stas: > > bq. using anything but none is not really an option > > > > If you have time, can you explain a bit more ? > > > > Thanks

Re: kafka broker loosing offsets?

If you set auto.offset.reset to none next time it happens you will be in much better position to find out what happens. Also in general with current semantics of offset reset policy IMO using anything but none is not really an option unless it is ok for consumer to loose some data (latest) or

Serve interactive queries from standby replicas

Hi Is there a way to serve read read requests from standby replicas? StreamsMeatadata does not seem to provide standby end points as far as I can see. Thank you, Stas

Re: kafka-streams configuration for high resilience (crashing when broker crashing) ?

I would set it to Integer.MAX_VALUE 2017-10-05 19:29 GMT+02:00 Dmitriy Vsekhvalnov <dvsekhval...@gmail.com>: > I see, but producer.retries set to 10 by default. > > What value would you recommend to survive random broker crashes? > > On Thu, Oct 5, 2017 at 8:24 PM

Re: kafka-streams configuration for high resilience (crashing when broker crashing) ?

mean by retires? I don't see retries property in StreamConfig > class. > > On Thu, Oct 5, 2017 at 7:55 PM, Stas Chizhov <schiz...@gmail.com> wrote: > > > Hi > > > > Have you set replication.factor and retries properties? > > > > BR > > >

Re: kafka-streams configuration for high resilience (crashing when broker crashing) ?

Hi Have you set replication.factor and retries properties? BR tors 5 okt. 2017 kl. 18:45 skrev Dmitriy Vsekhvalnov : > Hi all, > > we were testing Kafka cluster outages by randomly crashing broker nodes (1 > of 3 for instance) while still keeping majority of replicas

Re: Replication does not start because of OuOfOrderSequenceException

ds to be verified), we should > > consider whether a fix is needed for the 0.11.0 branch as well. > > > > Ismael > > > > On Mon, Oct 2, 2017 at 11:28 AM, Stas Chizhov <schiz...@gmail.com> > wrote: > > > > > Hi, > > > > > > We r

Replication does not start because of OuOfOrderSequenceException

Hi, We run 0.11.01 and there was a problem with 1 ReplicationFetcher on one of the brokers - it experience out of order sequence problem for one topic/partition and was stopped. It stayed stopped over the weekend. During this time log cleanup was working and by now it has cleaned up all the data

Re: windowed store excessive memory consumption

that do return iterators. > > That makes sense. Can you create a JIRA for this? Thanks. > > -Matthias > > On 9/27/17 2:54 PM, Stas Chizhov wrote: > > Thanks, that comment actually mad its way to the documentation already. > > Apparently none of that was related. It was a leak

Re: windowed store excessive memory consumption

CommentId=15984467=com.atlassian.jira. > plugin.system.issuetabpanels:comment-tabpanel#comment-15984467 > > On Wed, Sep 27, 2017 at 12:44 PM, Stas Chizhov <schiz...@gmail.com> wrote: > > > Hi, > > > > I am running a simple kafka streams app (0.11.0.1) that coun

windowed store excessive memory consumption

Hi, I am running a simple kafka streams app (0.11.0.1) that counts messages per hour per partition. The app runs in a docker container with a memory limit set, which is always reached by the app within few minutes and then container is killed. After running it with various number of instances,

KAFKA-5731

Hi, We are running confluent s3 conector (3.2.0) and we observed a sink task not being able to commit offsets after rebalance for like a week.It spits "WorkerSinkTask:337 - Ignoring invalid task provided offset -- partition not assigned" every time new file was written to S3. Eventually after 7

Metrics: committed offset, client version

Hi! I am wondering if there are broker/client metrics for: - client version (to keep track of clients that needs an upgrade) - committed offsets (to detect situations when commits fail systematically with everything else being ok) Thank you, Stanislav.

Kafka consumer: fair consumption from multiple partitions

Hi, I have to process a topic with few thousand messages and a dozen partitions from the very beginning. This topic is manually populated before consumption. In this setup a consumer consuming from several partitions at the same time tend to consume assigned partitions sequentially: first all

Re: Rename consumer group?

Hi, We've written a few-liner command that reads offsets for the consumer group we want to copy and commits those for a new group. That way you can inspect "__consumer_offsets" topic and make sure everything is correct before you start consuming messages. BR Stanislav. 2017-04-25 22:02

Re: Kafka source connector for avro files

Yes that should work. Thanks a lot! 2017-04-25 21:44 GMT+02:00 Gwen Shapira <g...@confluent.io>: > We added a Byte Converter which essentially does no conversion. Is this > what you are looking for? > > https://issues.apache.org/jira/browse/KAFKA-4783 > > On Tue, Apr 25

Kafka source connector for avro files

Hi, I have a kafka topic with avro messages + schema registry, which is being backed up into s3 as a set of avro files. I need to be able to restore a subset of those files into a new topic in the original format with schemas published into a schema registry. Am I right that at the moment there