Re: Where can I find the document for consumer metrics
On Thursday, February 11, 2016 at 19:53, Jason Gustafson wrote: > We have them in the Confluent docs: > http://docs.confluent.io/2.0.0/kafka/monitoring.html#new-consumer-metrics. Thanks Jason, that’s very helpful! Any chance this could be copied over to the canonical Kafka docs?
Re: Consumer backwards compatibility
When porting an existing consumer group from 0.8.2 to 0.9 (clients and brokers), is there any impact on the last committed offsets? is the "native offset storage" feature enable by default? On Thu, Feb 11, 2016 at 4:52 PM Jason Gustafson wrote: > The new Java consumer in 0.9.0 will not work with 0.8.2 since it depends on > the group management protocol built into Kafka, but the older consumer > should still work. > > -Jason > > On Thu, Feb 11, 2016 at 2:44 AM, Joe San wrote: > > > I have a 0.9.0 version of the Kafka consumer. Would that work against the > > 0.8.2 broker? > > >
Re: "Topic Created" event
Wonder if you can listen to the zkPath for topics via a zk watch ( https://zookeeper.apache.org/doc/r3.3.3/api/org/apache/zookeeper/Watcher.html) to let you know when the structure of the tree changes (ie, add/remove)? The zkPath for topics is "/brokers/topics" https://github.com/christian-posta/kafka/blob/trunk/core/src/main/scala/kafka/utils/ZkUtils.scala#L59-59 HTH! On Fri, Feb 12, 2016 at 9:07 AM, Karan Gupta wrote: > Hello, > > I am learning how to use kafka as my personal project. > I was wondering if there is a "topic created" event that kafka or zookeeper > sends such that a simple scala/java program can listen to. I was not able > to find any documentation on that. > > Thanks for any advice :) > -- > > Best regards > Karan > -- *Christian Posta* twitter: @christianposta http://www.christianposta.com/blog http://fabric8.io
"Topic Created" event
Hello, I am learning how to use kafka as my personal project. I was wondering if there is a "topic created" event that kafka or zookeeper sends such that a simple scala/java program can listen to. I was not able to find any documentation on that. Thanks for any advice :) -- Best regards Karan
Re: Connection to kafka stalls
Hi, An update about this. I've recreated the topic with different configuration and the problem doesn't seem to be happening anymore. I have 8 brokers. This topic was previously created (when the connections were stalling within the thread dump that I've attached) with 5 partitions, retetion policy of 5 days and replication factor of 2. The new configuration (which no longer causes this issue) has 8 partitions, same retention policy and same replication factor. What could have caused the connections to hang in the previous configuration? 2016-02-10 15:19 GMT-03:00 Maximiliano Patricio Méndez : > Sorry, I'm using kafka 0.8.2 and a ConsumerGroup similar to what it is in > the documentation. > > 2016-02-10 14:39 GMT-03:00 Maximiliano Patricio Méndez < > mmen...@despegar.com>: > >> Hi, >> >> I'm having trouble with some recurring stalling connections to kafka. >> What I see as a symptom is that some consumers lag behind and most times >> restarting the consumer doesn't help. (occasionally when some other >> consumer tries to take the problematic partition it no longer fails, but >> mostly even when it switches consumer it stalls shortly after). >> >> Doing a thread dump of this situation I see that the call stalls in the >> hasNext() method of the ConsumerIterator, although it has many messages to >> consume and that particular partition for that topic is lagged. >> >> "hermes-consumer-thread-1" #75 prio=5 os_prio=0 tid=0x7fe430fde000 >> nid=0x7c01 waiting on condition [0x7fe428ce1000] >>java.lang.Thread.State: TIMED_WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00070932c870> (a >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >> at >> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) >> at >> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) >> at >> kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:65) >> at >> kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33) >> at >> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66) >> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58) >> >> Reading through the mailing list I've come accross old solutions for this >> problem, including checking the consumer.timeout.ms (which i've added >> with no results) and checking the size of the messages (if the message is >> bigger than fetch.message.max.bytes it will stop like this) but my messages >> are all under 300 bytes in size. >> >> Have anyone had this problem? Any help would be appreciated >> >> Thanks >> >> >
Re: Kafka 0.8.2.0 Log4j
I have a logback.xml that I use when I do sbt run: sbt -Dlogback.configurationFile=app-logger.xml run On Fri, Feb 12, 2016 at 12:19 PM, Ben Stopford wrote: > Check you’re setting the Kafka log4j properties. > > -Dlog4j.configuration=file:config/log4j.properties > > B > > On 12 Feb 2016, at 07:33, Joe San wrote: > > > > How could I get rid of this warning? > > > > log4j:WARN No appenders could be found for logger > > (kafka.utils.VerifiableProperties). > > log4j:WARN Please initialize the log4j system properly. > > > > Any ideas how to get rid of this warning? > >
Re: Kafka 0.8.2.0 Log4j
But still could not get rid of this annoying warning! On Fri, Feb 12, 2016 at 12:58 PM, Joe San wrote: > I have a logback.xml that I use when I do sbt run: > > sbt -Dlogback.configurationFile=app-logger.xml run > > On Fri, Feb 12, 2016 at 12:19 PM, Ben Stopford wrote: > >> Check you’re setting the Kafka log4j properties. >> >> -Dlog4j.configuration=file:config/log4j.properties >> >> B >> > On 12 Feb 2016, at 07:33, Joe San wrote: >> > >> > How could I get rid of this warning? >> > >> > log4j:WARN No appenders could be found for logger >> > (kafka.utils.VerifiableProperties). >> > log4j:WARN Please initialize the log4j system properly. >> > >> > Any ideas how to get rid of this warning? >> >> >
Re: Kafka 0.8.2.0 Log4j
Check you’re setting the Kafka log4j properties. -Dlog4j.configuration=file:config/log4j.properties B > On 12 Feb 2016, at 07:33, Joe San wrote: > > How could I get rid of this warning? > > log4j:WARN No appenders could be found for logger > (kafka.utils.VerifiableProperties). > log4j:WARN Please initialize the log4j system properly. > > Any ideas how to get rid of this warning?
