Hi Kafka Users. Using kafka client and server of 0.11.0.2
I am trying trying to set a kafka consumer for long batching. I have tired small/short batches with settings like these and it seems to work just fine: poll timeout = 2000 max.poll.interval.ms = 50000 fetch.max.wait.ms = 500 fetch.min.bytes = 1 max.partition.fetch.bytes = 104857600 max.poll.records = 100 request.timeout.ms = 305000 But when trying to set a longer batch window we see Kafka get stuck from time to time, eg settings: poll timeout = 800000 max.poll.interval.ms = 850000 fetch.max.wait.ms = 720000 fetch.min.bytes = 200000 max.partition.fetch.bytes = 104857600 max.poll.records = 100 request.timeout.ms = 900000 Any thoughts on whether a long 10-12m batch window is feasible and if so are there both server and client settings to have to re-configure to ensure stability? Thoughts? -- -Michel
