Re: consumer timeout not being respected when kafka broker is not available

Just started testing consumers and can reproduce this, did you check JIRA or the forum. May be as this is consumer side, there isn't much you can do but wait right especially for commit Sync? On Thu, Jul 7, 2016 at 1:21 PM, Fumo, Vincent wrote: > KafkaConsumer v0.9:: > > I have a consumer set up

Re: Consumer Offsets and Open FDs

Try increasing log cleaner threads. On Tue, Jul 19, 2016 at 1:40 AM, Lawrence Weikum wrote: > It seems that the log-cleaner is still failing no matter what settings I > give it. > > Here is the full output from one of our brokers: > [2016-07-18 13:00:40,726] ERROR [kafka-log-cleaner-thread-0], E

Regarding kafka partition and replication

Hi, I have 3 nodes, each with 3 brokers, Kafka cluster along with 3 zookeeper cluster. So total 9 brokers spread across 3 different machines. I am adhered to Kafka 0.9. In order to optimally use the infrastructure for 2 topics (as of now, is not expected to grow drastically in near future), I am

Kafka Consumer stops consuming from a topic

Hi Everyone, Here are my settings Using Kafka 0.9.0.1, 1 instance (as we are testing things on a staging environment) Subscribing to 4 topics from a single Consumer application with 4 threads Now the server keeps on working fine for a while, then after about 3-4 hrs or so, it stops consuming at a

Re: Compacted topic cannot accept message without key

thanks for your answer，I know the necessity of key for compacted topics，and as you know，__consumer_offsets is a internal compacted topic in kafka，and it’s key is a triple of ，these errors are occurred when the consumer client wants to commit group offset. so why does his happen？ > 在 2016年7月19日

Kafka Streams Dynamic Topic consumer

Is there any way to specify a dynamic topic list (e.g. like a regex whitelist filter…like in the consumer API) with kafka streams? We would like to get the benefit of automatic checkpointing and load balancing if possible. -David

RE: TLS based ACL: Does Kafka support multiple CA Certs on broker

ca2root is in client truststorecaroot is not imported into client truststorekafka.example.com is not imported into client truststore oemkafka.example.com is not imported into client truststore Martin __ > From: ka...@harsha.io > Date: Mon, 18 Jul 20

Re: Producer Acks All vs -1

Hi Brian, Those settings actually mean the same thing . If you can come up with a better way to document it, it could make it clearer for others too :) On Mon, Jul 18, 2016

Re: Consumer Offsets and Open FDs

It seems that the log-cleaner is still failing no matter what settings I give it. Here is the full output from one of our brokers: [2016-07-18 13:00:40,726] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner) java.lang.IllegalArgumentException: requirement failed: 19205321

Producer Acks All vs -1

I am using version 0.10.0 of Kafka and the documentation syas the Producer acks can have the value can be [all, -1, 0, 1]. What is the difference between the all and -1 setting?

Question about a kafka use case : sequence once a partition is added

I want to use Kafka for notifications of changes to data in a dataservice/database. For each object that changes, a kafka message will be sent. This is easy and we've got that working no problem. Here is my use case : I want to be able to fire up a process that will 1) determine the current lo

Re: Deploying Kafka Streams to production (Ubuntu 16.04 / systemd)

This is something that perhaps someone else can comment on; I haven't played Kafka Streams yet. That said, your Kafka Streams application would need to shut down gracefully, whether you want to shut it down via Ctrl-C or use systemd :-). -Akshat On Mon, Jul 18, 2016 at 11:40 AM, Phillip Mann w

Re: Deploying Kafka Streams to production (Ubuntu 16.04 / systemd)

Hi Akshat, Do you know if a Kafka Streams apps can gracefully handle SIGTERM or do we have to implement it similarly to here: https://metabroadcast.com/blog/stop-your-spark-streaming-application-gracefully Thanks again for your help. Phillip From: Akshat Aranya Date: Monday, July 18, 2016 at

Questions about Kafka Streams Partitioning & Deployment

Hi all, My company, Taboola, has been looking at Kafka Streams and we are still confused about some details of partitions and store persistence. So let's say we have an app consuming a single topic and we're using an embedded and persisted key-value store: 1. If we restart the app, will each

RE: Kafka Fault Tolerance Test

Hi Guozhang, Thanks for your reply. Now the system is working fine. I did not use the right topic for testing, which only has one replica. Appreciated!! Best, Chao -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: Monday, July 18, 2016 1:17 PM To: users@kafka.ap

Re: Kafka Fault Tolerance Test

Also which version of Kafka are you using? On Mon, Jul 18, 2016 at 7:16 PM, Guozhang Wang wrote: > This is un-expected. Any error logs / exceptions did you see from the > clients when they can no longer send / fetch from brokers? > > Guozhang > > On Mon, Jul 18, 2016 at 8:59 AM, Luo, Chao wrote

Re: Kafka Fault Tolerance Test

This is un-expected. Any error logs / exceptions did you see from the clients when they can no longer send / fetch from brokers? Guozhang On Mon, Jul 18, 2016 at 8:59 AM, Luo, Chao wrote: > Dear Kafka fans, > > I have a concern of testing Kafka fault tolerance. Or may I did not > configure it r

Re: Compacted topic cannot accept message without key

Compacted topics require keyed messages in order for compaction to function. The solution is to provide a key for your messages. I would suggest reading the wiki on log compaction. On Mon, Jul 18, 2016 at 12:03 PM, Kafka wrote:

Re: Null Output topic for Kafka Streams

You can use KStream#foreach() as last operator. -Matthias On 07/18/2016 06:50 PM, David Garcia wrote: > I would like to process messages from an input topic, but I don’t want to > send messages downstream…with KStreams. (i.e. I would like to read from a > topic, do some processing including oc

Re: Deploying Kafka Streams to production (Ubuntu 16.04 / systemd)

Hi Phillip, In general, it is trivial to take an existing java -jar command and systemd-ify it. All you need to do is to run it as an ExecStart in the foreground in the systemd unit file, possibly with a User setting. Something like this should work: [Unit] Description=My stream app [Service] U

Null Output topic for Kafka Streams

I would like to process messages from an input topic, but I don’t want to send messages downstream…with KStreams. (i.e. I would like to read from a topic, do some processing including occasional database updates, and that’s it…no output to a topic). I could fake this by filtering out all my me

Re: Deploying Kafka Streams to production (Ubuntu 16.04 / systemd)

Hello Guozhang, Thanks for your reply. I am interested in systemd because that is the primary method for the environment we are working with (AWC EC2 Ubuntu 16.04 / systemd) at my company and on this project. If there is no way to deploy with systemd, then we can explore alternatives. Hopefu

Compacted topic cannot accept message without key

Hi, The server log shows error as belows on broker 0.9.0. ERROR [Replica Manager on Broker 0]: Error processing append operation on partition [__consumer_offsets,5] (kafka.server.ReplicaManager) kafka.message.InvalidMessageException: Compacted topic cannot accept message without

Kafka Fault Tolerance Test

Dear Kafka fans, I have a concern of testing Kafka fault tolerance. Or may I did not configure it right. I have two kafka servers and one zookeeper, which are running on three different AWS EC2 instances. I created a topic with one partition and two replica. First, the two kafka servers were r

SSL / SASL_SSL questions

Hi All, I have a question about the config I have working, and whether or not all traffic is being encrypted when sent via the client. Lets say I have the following settings, I'm only including the relevant parameters: Broker config: listeners=SASL_SSL://:9092,SSL://:9093 log.message.format.ve

Re: Chaining custom processors with DSL

Sure. You can use process(), transform(), or transformValues() on a KStream for general UDFs. See http://docs.confluent.io/3.0.0/streams/developer-guide.html#stateful-transformations -Matthias On 07/17/2016 11:56 PM, Srikanth wrote: > Hello, > > Using the low level API its possible to chain p

Out of sync replica

Hi, I got an issue with our kafka nodes. We replaced some nodes on the cluster and reassigned the partitions from a new topic layout. After reassigning, zookeeper lists the correct broker ids, but kafka stuck with its older brokers in its replicas. In the example below, the leader is a valid

Helping Spread the Word about Apachecon EU 2016

Hi Everyone I'm forwarding the following message on behalf of Rich Bowen and the Apachecon team == As you are aware, we are holding ApacheCon in Seville in November. While this seems like a long way away, it is critical that we get on people's calendar now, so that they can plan, get b