Re: JMX Port Conflict when starting both Broker and kafka Connect

2018-11-30 Thread Sruthi Kumar Annamneedu
Hi Harish, Using netstat command, see which process is using the port. See if following steps help: - If processA is intended to use portX which is having issues, stop processA - If the processB is using the port, then stop ProcessB - Restart processB and try using the port - Restart the proc

Re: Partial updates in Kafka Streams stores

2018-11-30 Thread Matthias J. Sax
Yes, that could happen. If you write into a store, the write _might_ go directly into the changelog topic, too, and for this case, the instance that takes over would see this write and add it to it's store during recovery. Idempotent updates would solve this issue. -Matthias On 11/30/18 12:17 PM

Re: Partial updates in Kafka Streams stores

2018-11-30 Thread Yoshimo
Hello Matthias, thank you for the answer. What if the operations on the stores are idempotent? I am only worried for the scenario where after doing a put in the first store and before updating the second one, the application crashes or there is a network outage. Will the node that picks up the

Re: Partial updates in Kafka Streams stores

2018-11-30 Thread Matthias J. Sax
You can enable exactly-once processing guarantees to guard against inconsistent stores in fail over scenario. -Matthias On 11/30/18 12:57 AM, Yoshimo wrote: > Hello Kafka users, > > I am currently building a Kafka Streams application and I am using a > transform step with two KeyValue stores,

Partial updates in Kafka Streams stores

2018-11-30 Thread Yoshimo
Hello Kafka users, I am currently building a Kafka Streams application and I am using a transform step with two KeyValue stores, one containing the transformed message and the second one some metadata about the message. In the transform step I first update the metadata store and then store the

JMX Port Conflict when starting both Broker and kafka Connect

2018-11-30 Thread harish lohar
Hi , I am trying to use kafkaConnect for mongdb-source-connector I am using connect-distributed.sh. When i start kafka with KAFKA_JMX_OPTS, kafka broker starts fine but as soon as kafka connect is started it gives javaagent:./jmx_prometheus_javaagent-0.6.jar=7071:./kafka-0-8-2.yml" address alre

Setup GC log rotate policy for kafka broker servers

2018-11-30 Thread Parth Gandhi
Team, I took over kafka from a colleague and found that the GC log has been setup without setting up the log rotate policy. Now the GC log size has reached to 4 GB and we need to move this file and also set up a log rotate policy. Can this be achieved without restarting the kafak service? We have

Re: Please explain Rest API

2018-11-30 Thread Sönke Liebau
Hi Satendra, in what order did you start your connector and the console consumer and did you specify "--from-beginning"? >From your connector config it looks like the connector will only poll for data every 15 minutes (90 ms). So if you started the connector first and then your consumer it mig

Dead letter queue / quarantined topic

2018-11-30 Thread Bruno Bieth
We are building a kafka-streams application as part of a large microservices architecture. We want to be resilient to backward incompatible format changes and have introduced a quarantined topic. We couldn't find anything provided by the library so we sort of rolled our own, by simply "manually" tr

Re: Please explain Rest API

2018-11-30 Thread jan
I may have missed this (I'm missing the first few messages), so sorry in advance if I have, but what OS are you using? Kafka does not work well on windows, I had problems using it that sounds a little like this (just a little though) when on win. jan On 30/11/2018, Satendra Pratap Singh wrote: >