[jira] [Commented] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
[ https://issues.apache.org/jira/browse/STORM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098242#comment-15098242 ] Garrick Dasbach commented on STORM-1471: Not sure how far out the JStorm port is, but in the interim, maybe it would be sufficient to make the Kafka fetch asynchronous. That way we can allow long-polling without blocking the spout. > Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig > - > > Key: STORM-1471 > URL: https://issues.apache.org/jira/browse/STORM-1471 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Reporter: Garrick Dasbach > > We currently have an issue in our storm cluster where our Kafka brokers are > under heavy load due to too many fetch requests from storm. We've narrowed > the problem to the way Fetch Requests are build in KafkaUtils. When using > the FetchRequestBuilder, storm provides overrides for all the properties > except minBytes. The default for that field is 0 (even though the Kafka > default for the high-level consumer is 1). When paired with a maxWait > 0, > this creates a situation where the broker can immediately return a response > without waiting (due to minBytes 0). This puts a heavy load on the brokers > and defeats the purpose of any long polling. > By making this a SpoutConfig option, it will allow the user to set that as > appropriate for their situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
[ https://issues.apache.org/jira/browse/STORM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097475#comment-15097475 ] Garrick Dasbach commented on STORM-1471: That's a fair point, however it's somewhat confusing then that the KafkaConfig allows you to set fetchMaxWait, which defaults to 10s. > Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig > - > > Key: STORM-1471 > URL: https://issues.apache.org/jira/browse/STORM-1471 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Reporter: Garrick Dasbach > > We currently have an issue in our storm cluster where our Kafka brokers are > under heavy load due to too many fetch requests from storm. We've narrowed > the problem to the way Fetch Requests are build in KafkaUtils. When using > the FetchRequestBuilder, storm provides overrides for all the properties > except minBytes. The default for that field is 0 (even though the Kafka > default for the high-level consumer is 1). When paired with a maxWait > 0, > this creates a situation where the broker can immediately return a response > without waiting (due to minBytes 0). This puts a heavy load on the brokers > and defeats the purpose of any long polling. > By making this a SpoutConfig option, it will allow the user to set that as > appropriate for their situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
[ https://issues.apache.org/jira/browse/STORM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097329#comment-15097329 ] Jungtaek Lim commented on STORM-1471: - AFAIK, It is intended behavior since nextTuple() in Spout shouldn't be blocked. You can refer http://storm.apache.org/documentation/Concepts.html - Spouts section. JStorm provides multi-thread mode of Spout, and it could be applied to Storm when merging Storm and JStorm. https://issues.apache.org/jira/browse/STORM-1358 > Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig > - > > Key: STORM-1471 > URL: https://issues.apache.org/jira/browse/STORM-1471 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Reporter: Garrick Dasbach > > We currently have an issue in our storm cluster where our Kafka brokers are > under heavy load due to too many fetch requests from storm. We've narrowed > the problem to the way Fetch Requests are build in KafkaUtils. When using > the FetchRequestBuilder, storm provides overrides for all the properties > except minBytes. The default for that field is 0 (even though the Kafka > default for the high-level consumer is 1). When paired with a maxWait > 0, > this creates a situation where the broker can immediately return a response > without waiting (due to minBytes 0). This puts a heavy load on the brokers > and defeats the purpose of any long polling. > By making this a SpoutConfig option, it will allow the user to set that as > appropriate for their situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
