Way to check if custom SMT has been added to the classpath or even if it i working.
Hi All, i have created a custom SMT and have deployed. I would like to know if there is a way to check if the transform is working or not.(def not working as the messages are not getting transformed) I am also trying to remote debug using intellij and nothing seam working, as i do not see any control hitting the debug points. When i check the connector list using , curl localhost:8083/connector-plugins , i see all other connector plugins but not the SMT related ones. Regards.
Build error using intellij.
Hi, I am trying to build kafka from source code, but i get below error when i try to build the project.(Have use gradle .idea command) and when i try to click on the import statements, they end up opening in test folder of client package and not main package. Any help would be appreciated. /Users/svegesna/svegesna/kafka/log4j-appender/src/main/java/org/apache/kafka/log4jappender/KafkaLog4jAppender.java Error:(19, 41) java: package org.apache.kafka.clients.producer does not exist Error:(20, 41) java: package org.apache.kafka.clients.producer does not exist Error:(21, 41) java: package org.apache.kafka.clients.producer does not exist Error:(22, 41) java: package org.apache.kafka.clients.producer does not exist Error:(23, 38) java: package org.apache.kafka.common.config does not exist Error:(24, 45) java: package org.apache.kafka.common.serialization does not exist Error:(35, 48) java: package org.apache.kafka.clients.producer does not exist Error:(35, 1) java: static import only from classes and interfaces Error:(36, 48) java: package org.apache.kafka.clients.producer does not exist Error:(36, 1) java: static import only from classes and interfaces Error:(37, 48) java: package org.apache.kafka.clients.producer does not exist Error:(37, 1) java: static import only from classes and interfaces Error:(38, 48) java: package org.apache.kafka.clients.producer does not exist Error:(38, 1) java: static import only from classes and interfaces Error:(39, 48) java: package org.apache.kafka.clients.producer does not exist Error:(39, 1) java: static import only from classes and interfaces Error:(40, 39) java: package org.apache.kafka.clients does not exist Error:(40, 1) java: static import only from classes and interfaces Error:(41, 39) java: package org.apache.kafka.clients does not exist Error:(41, 1) java: static import only from classes and interfaces Error:(42, 45) java: package org.apache.kafka.common.config does not exist Error:(42, 1) java: static import only from classes and interfaces Error:(43, 45) java: package org.apache.kafka.common.config does not exist Error:(43, 1) java: static import only from classes and interfaces Error:(44, 45) java: package org.apache.kafka.common.config does not exist Error:(44, 1) java: static import only from classes and interfaces Error:(45, 45) java: package org.apache.kafka.common.config does not exist Error:(45, 1) java: static import only from classes and interfaces Error:(46, 45) java: package org.apache.kafka.common.config does not exist Error:(46, 1) java: static import only from classes and interfaces Error:(47, 45) java: package org.apache.kafka.common.config does not exist Error:(47, 1) java: static import only from classes and interfaces Error:(70, 13) java: cannot find symbol symbol: class Producer location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(72, 12) java: cannot find symbol symbol: class Producer location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(242, 15) java: cannot find symbol symbol: class Producer location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(201, 23) java: cannot find symbol symbol: variable BOOTSTRAP_SERVERS_CONFIG location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(203, 23) java: cannot find symbol symbol: class ConfigException location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(205, 23) java: cannot find symbol symbol: class ConfigException location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(207, 23) java: cannot find symbol symbol: variable COMPRESSION_TYPE_CONFIG location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(209, 23) java: cannot find symbol symbol: variable ACKS_CONFIG location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(211, 23) java: cannot find symbol symbol: variable RETRIES_CONFIG location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(213, 23) java: cannot find symbol symbol: variable SECURITY_PROTOCOL_CONFIG location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(217, 23) java: cannot find symbol symbol: variable SSL_TRUSTSTORE_LOCATION_CONFIG location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(218, 23) java: cannot find symbol symbol: variable SSL_TRUSTSTORE_PASSWORD_CONFIG location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(222, 27) java: cannot find symbol symbol: variable SSL_KEYSTORE_TYPE_CONFIG location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(223, 27) java: cannot find symbol symbol: variable SSL_KEYSTORE_LOCATION_CONFIG location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(224, 27) java: cannot find symbol symbol: variable SSL_KEYSTORE_PASSWORD_CONFIG location: class org.apache.kafka.log4jappender.KafkaLog4jAppender Error:(228, 23) java: cannot find
ZkUtils.getAllPartitions giving more partition counts.
Hi, Would like to understand the purpose of ZkUtils.getAllPartitions, as when i try to use the method, i end up getting wrong number of partitions assigned to Topics, i am not really sure if my understanding is wrong about this method. i have assumed this method would return the partition count. But i end up getting Set[TopicAndPartition] objects, and when i try to sum up the count of partitions, for a single topic, they exceed from the actual topic count. Regards, Satyajit.
Kafka Consumers not rebalancing.
Hi All, I was experimenting on the new consumer API and have a question regarding the rebalance process. I start a consumer group with single thread and make the Thread sleep while processing the records retrieved from the first consumer.poll call, i was making sure the Thread.sleep time goes beyond the session timeout and was expecting to see a rebalance on the consumer group. But when i try to get the state of the consumer group using the below command, /opt/confluent-4.0.0/bin/kafka-consumer-groups --group "consumer-grievances-group15" --bootstrap-server xxx:9092,:9092,:9092 --describe i get below result , Consumer group 'consumer-grievances-group15' has no active members. TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAGCONSUMER-ID HOST CLIENT-ID TELMATEQA.grievances.grievances 0 125855 152037 26182 - - - The same happens with multiple thread in the consumer group scenario, and *going further one step i was able to make the thread get into running state, from sleep state and could see that the consumer group started off from where it left.* My only question is , why isn't the rebalancing happening in this scenario. My expectation was that the threads rebalance and start from the committed offset. Regards, Satyajit.
Re: Kafka Consumers not rebalancing.
Any help would be appreciated! Hi All, I was experimenting on the new consumer API and have a question regarding the rebalance process. I start a consumer group with single thread and make the Thread sleep while processing the records retrieved from the first consumer.poll call, i was making sure the Thread.sleep time goes beyond the session timeout and was expecting to see a rebalance on the consumer group. But when i try to get the state of the consumer group using the below command, /opt/confluent-4.0.0/bin/kafka-consumer-groups --group "consumer-grievances-group15" --bootstrap-server xxx:9092,:9092, :9092 --describe i get below result , Consumer group 'consumer-grievances-group15' has no active members. TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAGCONSUMER-ID HOST CLIENT-ID TELMATEQA.grievances.grievances 0 125855 152037 26182 - - - The same happens with multiple thread in the consumer group scenario, and *going further one step i was able to make the thread get into running state, from sleep state and could see that the consumer group started off from where it left.* My only question is , why isn't the rebalancing happening in this scenario. My expectation was that the threads rebalance and start from the committed offset. Regards, Satyajit. On Mon, Jan 29, 2018 at 3:36 PM, satyajit vegesna wrote: > Hi All, > > I was experimenting on the new consumer API and have a question regarding > the rebalance process. > > I start a consumer group with single thread and make the Thread sleep > while processing the records retrieved from the first consumer.poll call, i > was making sure the Thread.sleep time goes beyond the session timeout and > was expecting to see a rebalance on the consumer group. > > But when i try to get the state of the consumer group using the below > command, > > /opt/confluent-4.0.0/bin/kafka-consumer-groups --group > "consumer-grievances-group15" --bootstrap-server xxx:9092,:9092, > :9092 --describe > > i get below result , > > Consumer group 'consumer-grievances-group15' has no active members. > > > TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET > LAGCONSUMER-ID HOST > CLIENT-ID > > TELMATEQA.grievances.grievances 0 125855 152037 > 26182 - - > - > > The same happens with multiple thread in the consumer group scenario, and > *going > further one step i was able to make the thread get into running state, from > sleep state and could see that the consumer group started off from where it > left.* > > My only question is , why isn't the rebalancing happening in this > scenario. My expectation was that the threads rebalance and start from the > committed offset. > > Regards, > Satyajit. > >
