zhangyouxun commented on issue #11061: URL: https://github.com/apache/druid/issues/11061#issuecomment-811954109
> Your task count is 30, does that mean there're 30 partitions of your topic ? And is the ingestion speed 5K/s of per peon ? Even it's 5K/s per peon, it's not a very high speed that it needs 10G heap size for each peon. 4G is enough in our clusters to handle messages at much higher speed. > > Since streaming ingestion speed is more relevant to streaming middleware, tuning producer and consumer may help. Are you using Kafka ? And what parameters you have adopted to configure Kafka producers and consumers ? the num of partitions is 270. The maxBytesInMemory is about 300Mb, is it not make full use of heap? we use Kafka max.poll.records = 500 max.partition.fetch.bytes = 1048576 fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 metadata.max.age.ms = 10000 which parameter needs to be adjusted？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
