Re: Kafka broker slow down when consumer try to fetch large messages from topic
Hi Thanks for the information . I have one question : Right now in my scenario maximum message size is around 800KB . did we consider these messages in large size categories , because article told about 10-100 MB data . Regards Prateek On Thu, Jun 2, 2016 at 6:54 AM, Tom Crayford wrote: > Hi there, > > Firstly, a note that Kafka isn't really designed for this kind of large > message. http://ingest.tips/2015/01/21/handling-large-messages-kafka/ > covers a lot of tips around this use case however, and covers some tuning > that will likely improve your usage. > > In particular, I expect tuning up fetch.message.max.bytes on the consumer > to help out a lot here. > > Generally though, doing large messages will lead to very low throughput and > lots of stability issues, as noted in that article. We run thousands of > clusters in production, and typically recommend folk keep message sizes > down to the few tens of KB for most use cases. > > Thanks > > Tom Crayford > Heroku Kafka > > On Wed, Jun 1, 2016 at 9:49 PM, prateek arora > wrote: > > > I have 4 node kafka broker with following configuration : > > > > Default Number of Partitions : num.partitions : 1 > > Default Replication Factor : default.replication.factor : 1 > > Maximum Message Size : message.max.bytes : 10 MB > > Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB > > > > > > Right now I have 4 topic with 1 partition and 1 replication factor . > > > > "Topic Name" : "Broker Id" : "Total Messages Received Across Kafka > > Broker" : "Total Bytes Received Across Kafka Broker" > > Topic 1 - Leader Kafka Broker 1 : 4.67 Message/Second : 1.6 MB/second > > Topic 2 - Leader Kafka Broker 2 : 4.78 Message/Second : 4.1 MB/second > > Topic 3 - Leader Kafka Broker 1 : 4.83 Message/Second : 1.6 > MB/second > > Topic 4 - Leader Kafka Broker 3 : 4.8 Message/Second : 4.3 > MB/second > > > > Message consist of . > > > > > > when consumer tried to read message from "Topic 2" Kafka Broker rate of > > message receiving slow down from 4.77 message/second to 3.12 > > message/second , after some time try to goes up . > > > > I also attached screenshot of "Total Messages Received Across Kafka > > Broker" and "Total Bytes Received Across Kafka Broker" for topic "Topic > > 2" . > > > > can someone explain why it is happen and how to solve it ? > > > > Regards > > Prateek > > > > > > > > > > > > > > >
Kafka broker slow down when consumer try to fetch large messages from topic
I have 4 node kafka broker with following configuration : Default Number of Partitions : num.partitions : 1 Default Replication Factor : default.replication.factor : 1 Maximum Message Size : message.max.bytes : 10 MB Replica Maximum Fetch Size : replica.fetch.max.bytes : 10 MB Right now I have 4 topic with 1 partition and 1 replication factor . "Topic Name" : "Broker Id" : "Total Messages Received Across Kafka Broker" : "Total Bytes Received Across Kafka Broker" Topic 1 - Leader Kafka Broker 1 : 4.67 Message/Second : 1.6 MB/second Topic 2 - Leader Kafka Broker 2 : 4.78 Message/Second : 4.1 MB/second Topic 3 - Leader Kafka Broker 1 : 4.83 Message/Second : 1.6 MB/second Topic 4 - Leader Kafka Broker 3 : 4.8 Message/Second : 4.3 MB/second Message consist of . when consumer tried to read message from "Topic 2" Kafka Broker rate of message receiving slow down from 4.77 message/second to 3.12 message/second , after some time try to goes up . I also attached screenshot of "Total Messages Received Across Kafka Broker" and "Total Bytes Received Across Kafka Broker" for topic "Topic 2" . can someone explain why it is happen and how to solve it ? Regards Prateek
How can i secure Kafka topics
Hi Can i secure Kafka topics like Apache hbase and hdfs have kerberos security mechanism ? i am not able to find out any document related to Kafka security . please suggest how can i secure Kafka . Regards Prateek
Re: Facing Issue to create asyn producer in kafka 0.8.2
Thanks its work ... Regards Prateek On Wed, Oct 14, 2015 at 11:46 AM, Grant Henke wrote: > Looks like you may be mixing the new producer with old producer configs. > See the new config documentation here: > http://kafka.apache.org/documentation.html#newproducerconfigs. You will > likely want to set the "batch.size" and "linger.ms" to achieve your goal. > > Thanks, > Grant > > On Wed, Oct 14, 2015 at 1:29 PM, prateek arora > > wrote: > > > Hi > > > > Thanks for help . > > > > but same behavior even after changing batch.size > > > > I have changes batch.size value to 33554432. > > props.put("batch.size","33554432"); > > > > > > > > On Wed, Oct 14, 2015 at 11:09 AM, Zakee wrote: > > > > > Hi Prateek, > > > > > > Looks like you are using default batch.size which is ~16K and it forces > > > the send of messages immediately as your single message is larger than > > > that. Try using larger batch.size. > > > > > > Thanks > > > Zakee > > > > > > > > > > > > > On Oct 14, 2015, at 10:29 AM, prateek arora < > > prateek.arora...@gmail.com> > > > wrote: > > > > > > > > Hi > > > > > > > > I want to create async producer so i can buffer messages in queue and > > > send > > > > after every 5 sec . > > > > > > > > my kafka version is 0.8.2.0. > > > > > > > > and i am using kafka-clients 0.8.2.0 to create kafka producer in > java. > > > > > > > > > > > > below is my sample code : > > > > > > > > package com.intel.labs.ive.cloud.testKafkaProducerJ; > > > > > > > > import java.nio.charset.Charset; > > > > import java.util.HashMap; > > > > > > > > import java.util.Map; > > > > > > > > import org.apache.kafka.clients.producer.KafkaProducer; > > > > import org.apache.kafka.clients.producer.Producer; > > > > import org.apache.kafka.clients.producer.ProducerConfig; > > > > import org.apache.kafka.clients.producer.ProducerRecord; > > > > import org.apache.kafka.common.Metric; > > > > import org.apache.kafka.common.MetricName; > > > > import org.apache.kafka.common.serialization.Serializer; > > > > import org.apache.kafka.common.serialization.StringSerializer; > > > > import org.apache.kafka.common.serialization.ByteArraySerializer; > > > > > > > > import java.nio.file.DirectoryStream; > > > > import java.nio.file.Files; > > > > import java.nio.file.Path; > > > > import java.nio.file.Paths; > > > > > > > > public class TestKafkaProducer { > > > > > > > > Map props = new HashMap(); > > > >props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, > > > metadataBroker); > > > >props.put("producer.type", "async"); > > > >props.put("queue.buffering.max.ms", "5000"); > > > > > > > > Serializer keySerializer = new StringSerializer(); > > > >Serializer valueSerializer = new > ByteArraySerializer(); > > > > > > > >producer = new KafkaProducer(props, > > keySerializer, > > > > valueSerializer); > > > > > > > > ProducerRecord imageRecord; > > > > > > > > while ( true ) { > > > > imageRecord = new ProducerRecord(topicName, > > > > recordKey,imageBytes); > > > > > > > >producer.send(imageRecord); > > > > } > > > > } > > > > > > > > size of my message is around 77K > > > > > > > > but its work like a synchronous producer , send every message to > broker > > > . > > > > not buffering a message in to queue and send after 5 sec > > > > > > > > > > > > please help to find out a solution. > > > > > > > > > > > > Regards > > > > Prateek > > > > > > > > > A Balance Transfer Card With An Outrageously Long Intro Rate And No > > > Balance Transfer Fees That Can Save You Thousands > > > > http://thirdpartyoffers.netzero.net/TGL3231/561e9a75a77071a74763fst04vuc > > > > > > -- > Grant Henke > Software Engineer | Cloudera > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke >
Re: Facing Issue to create asyn producer in kafka 0.8.2
Hi Thanks for help . but same behavior even after changing batch.size I have changes batch.size value to 33554432. props.put("batch.size","33554432"); On Wed, Oct 14, 2015 at 11:09 AM, Zakee wrote: > Hi Prateek, > > Looks like you are using default batch.size which is ~16K and it forces > the send of messages immediately as your single message is larger than > that. Try using larger batch.size. > > Thanks > Zakee > > > > > On Oct 14, 2015, at 10:29 AM, prateek arora > wrote: > > > > Hi > > > > I want to create async producer so i can buffer messages in queue and > send > > after every 5 sec . > > > > my kafka version is 0.8.2.0. > > > > and i am using kafka-clients 0.8.2.0 to create kafka producer in java. > > > > > > below is my sample code : > > > > package com.intel.labs.ive.cloud.testKafkaProducerJ; > > > > import java.nio.charset.Charset; > > import java.util.HashMap; > > > > import java.util.Map; > > > > import org.apache.kafka.clients.producer.KafkaProducer; > > import org.apache.kafka.clients.producer.Producer; > > import org.apache.kafka.clients.producer.ProducerConfig; > > import org.apache.kafka.clients.producer.ProducerRecord; > > import org.apache.kafka.common.Metric; > > import org.apache.kafka.common.MetricName; > > import org.apache.kafka.common.serialization.Serializer; > > import org.apache.kafka.common.serialization.StringSerializer; > > import org.apache.kafka.common.serialization.ByteArraySerializer; > > > > import java.nio.file.DirectoryStream; > > import java.nio.file.Files; > > import java.nio.file.Path; > > import java.nio.file.Paths; > > > > public class TestKafkaProducer { > > > > Map props = new HashMap(); > >props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, > metadataBroker); > >props.put("producer.type", "async"); > >props.put("queue.buffering.max.ms", "5000"); > > > > Serializer keySerializer = new StringSerializer(); > >Serializer valueSerializer = new ByteArraySerializer(); > > > >producer = new KafkaProducer(props, keySerializer, > > valueSerializer); > > > > ProducerRecord imageRecord; > > > > while ( true ) { > > imageRecord = new ProducerRecord(topicName, > > recordKey,imageBytes); > > > >producer.send(imageRecord); > > } > > } > > > > size of my message is around 77K > > > > but its work like a synchronous producer , send every message to broker > . > > not buffering a message in to queue and send after 5 sec > > > > > > please help to find out a solution. > > > > > > Regards > > Prateek > > > A Balance Transfer Card With An Outrageously Long Intro Rate And No > Balance Transfer Fees That Can Save You Thousands > http://thirdpartyoffers.netzero.net/TGL3231/561e9a75a77071a74763fst04vuc
Facing Issue to create asyn producer in kafka 0.8.2
Hi I want to create async producer so i can buffer messages in queue and send after every 5 sec . my kafka version is 0.8.2.0. and i am using kafka-clients 0.8.2.0 to create kafka producer in java. below is my sample code : package com.intel.labs.ive.cloud.testKafkaProducerJ; import java.nio.charset.Charset; import java.util.HashMap; import java.util.Map; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.Producer; import org.apache.kafka.clients.producer.ProducerConfig; import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.common.Metric; import org.apache.kafka.common.MetricName; import org.apache.kafka.common.serialization.Serializer; import org.apache.kafka.common.serialization.StringSerializer; import org.apache.kafka.common.serialization.ByteArraySerializer; import java.nio.file.DirectoryStream; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; public class TestKafkaProducer { Map props = new HashMap(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, metadataBroker); props.put("producer.type", "async"); props.put("queue.buffering.max.ms", "5000"); Serializer keySerializer = new StringSerializer(); Serializer valueSerializer = new ByteArraySerializer(); producer = new KafkaProducer(props, keySerializer, valueSerializer); ProducerRecord imageRecord; while ( true ) { imageRecord = new ProducerRecord(topicName, recordKey,imageBytes); producer.send(imageRecord); } } size of my message is around 77K but its work like a synchronous producer , send every message to broker . not buffering a message in to queue and send after 5 sec please help to find out a solution. Regards Prateek
facing issue in async producer application
Hi I want to create producer in async mode so i can send message in 5 sec interval. below is my code : Map props = new HashMap(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, metadataBroker); // props.put(ProducerConfig.ACKS_CONFIG, "0"); // don't wait for ACKs props.put("producer.type", "async"); // don't wait for ACKs props.put("queue.buffering.max.ms", "5000"); // don't wait for ACKs Serializer keySerializer = new StringSerializer(); Serializer valueSerializer = new ByteArraySerializer(); producer = new KafkaProducer(props, keySerializer, valueSerializer); ProducerRecord imageRecord; imageRecord = new ProducerRecord(topicName, recordKey,imageBytes); producer.send(imageRecord);