Greets To team with a issue .
Hi Team , First of all my warm regards to each and everyone in the community . We are using kafka for centralizing the logging through our php application . Earlier our application was writing all the logs in a file in json line format , which we used to read using filebeat ( ELK family member ) and send it to logstash instace which in turn send it to kafka topics using kafka output plugin . Recently we enhanced our logger and made a producer which logs directly to kafka server using a kafka handler ( This handler uses phpkafka ( on the top of librdkafka) . We ran into situation where if broker was down the php application was not able to produce any more logs throwing error . The error which We recieved at consumer end was [2016-03-02 15:25:18,233] ERROR [ConsumerFetcherThread-console-consumer-14821_zk1-1456912451600-6bccbb09-0-1], Error for partition [cron,1] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) We have 3 zookeeper servers and 4 kafka brokers . I have topic with 5 partition and 3 replicas each . Now if 1 of the broker gets down , application should automatically detect leader and start writing in the available leader . But application ends with segementation fault . Please help . Regards Arvind Sharma Software Engg. Shopclues
Greets To team with a issue
Hi Team , First of all my warm regards to each and everyone in the community . We are using kafka for centralizing the logging through our php application . Earlier our application was writing all the logs in a file in json line format , which we used to read using filebeat ( ELK family member ) and send it to logstash instace which in turn send it to kafka topics using kafka output plugin . Recently we enhanced our logger and made a producer which logs directly to kafka server using a kafka handler ( This handler uses phpkafka ( on the top of librdkafka) . We ran into situation where if broker was down the php application was not able to produce any more logs throwing error . The error which We recieved at consumer end was [2016-03-02 15:25:18,233] ERROR [ConsumerFetcherThread-console-consumer-14821_zk1-1456912451600-6bccbb09-0-1], Error for partition [cron,1] to broker 1:class kafka.common.NotLeaderForPartitionException (kafka.consumer.ConsumerFetcherThread) We have 3 zookeeper servers and 4 kafka brokers . I have topic with 5 partition and 3 replicas each . Now if 1 of the broker gets down , application should automatically detect leader and start writing in the available leader . But application ends with segementation fault . Please help . Regards Arvind Sharma Software Engg. Shopclues
Strategic help in benchmarkig Kafka brokers
Hello All, We are trying to load test Kafka Cluster but we are not able to come up with exact maths for producers concurrency , partition number, replication factor , number of broker I am just trying to find out what is the maximum concurrent connection a partition of 1 node Kafka broker can adhere , then gradually move to increase partition on same node and get numbers how many partitions a node can sustain After that add broker to cluster and repeat the same test again. Is their any reference to load test in this scenario. Regards
Re: Apache Kafka training
If you like to study online then there's a series by Stephane Maarek on udemy , till now it's one of the best content I ever saw . Btw I am not promoting either of the 2 (udemy or Stephane ) in any way Regards On Thu, 26 Sep 2019, 3:59 pm Sachin Bal, wrote: > Hi All, > > Can someone suggest a trainer for Apache Kafka who can train on all aspects > of Kafka from beginner to advanced? Would be great, if there is some > training in Mumbai. > > Thanks > Sachin >
