failed with LeaderNotAvailableError -

2015-12-16 Thread David Montgomery
Hi, I am very concerned about using kafka in production given the below errors: Now issues with myt zookeeper. Other services use ZK. Only kafka fails. I have 2 kafka servers using 8.x. How do I resolve? I tried restarting services for kafka. Below is my kafka server.properties file 'Traceb

Re: kafka-connect-jdbc: ids, timestamps, and transactions

2015-12-16 Thread Ewen Cheslack-Postava
Mark, There are definitely limitations to using JDBC for change data capture. Using a database-specific implementation, especially if you can read directly off the database's log, will be able to handle more situations like this. Cases like the one you describe are difficult to address efficiently

Low-latency, high message size variance

2015-12-16 Thread Jens Rantil
Hi again Jason, Once again thanks for your answer. Yes, the more I think/read about this it sounds like the "max records" approach is more viable. Without knowing the code, I guess it would make more sense to create a "max.partition.fetch.messages" property. That way a consumer could optimize for

Re: Low-latency, high message size variance

2015-12-16 Thread Jens Rantil
Hi again Jason, Sorry for a bit of a late response - I'm travelling and check my e-mail spuriously. I have a specific question regarding they pause solution quoted below: On Tuesday, December 15, 2015, Jason Gustafson wrote: > > while (running) { > ConsumerRecords records = consumer.poll(1000

reassign __consumer_offsets partitions

2015-12-16 Thread Damian Guy
Hi, We have had some temporary nodes in our kafka cluster and i now need to move assigned partitions off of those nodes onto the permanent members. I'm familiar with the kafka-reassign-partitions script, but ... How do i get it to work with the __consumer_offsets partition? It currently seems to

kafka-connect-jdbc: ids, timestamps, and transactions

2015-12-16 Thread Mark Drago
I had asked this in a github issue but I'm reposting here to try and get an answer from a wider audience. Has any thought gone into how kafka-connect-jdbc will be impacted by SQL transactions committing IDs and timestamps out-of-order? Let me give an example with two connections. 1: begin transa

Consumer lag JMX in 0.9

2015-12-16 Thread Marko Bonaći
Hi, there was a problem with JMX consumer lag in 0.8: http://search-hadoop.com/m/uyzND14v72215XZpK&subj=Re+Consumer+lag+lies+orphaned+offsets+ Has anything changed now with 0.9? Thanks

Re: Low-latency, high message size variance

2015-12-16 Thread Ismael Juma
Jason, maybe useful to KAFKA-2986 with this information if we ever decide to do this? Ismael On 16 Dec 2015 04:42, "Jason Gustafson" wrote: > I was talking with Jay this afternoon about this use case. The tricky thing > about adding a ping() or heartbeat() API is that you have to deal with the >