for rate limiting, would quota at Kafka brokers help? On Tue, Nov 6, 2018 at 10:29 AM Ning Shi <nings...@gmail.com> wrote:
> On Tue, Nov 06, 2018 at 07:44:50PM +0200, Nicos Maris wrote: > > Ning can you provide another example except for rate limiting? > > Our main use case and concern is rate limiting because without it we > could potentially overwhelm downstream systems (Cassandra) when the job > plays catch up or replay events from Kafka. The consequence of that is > that exceptions will be thrown in the Cassandra sink causing the whole > job to restart and end up in the same situation over and over. > > With that said, we do have other use cases such as doing canary > deployment. We'd like to start processing events for a subset of users, > then expand it to all if things look good. Without live configuration, > we have to take a savepoint and restart the job, which will cause the > job to play catch up at the beginning, potentially overwhelming > downstream system without rate limiting. > > Hope the use cases described above clarifies. > > Thanks, > > -- > Ning >
