Recommended module for connector ratelimiting?

Hi all, The PR that I've been iterating on w.r.t. Kafka ratelimiting — https://github.com/apache/flink/pull/7679 is now in a state where a lot of the rate limiting logic is generic and does not apply specifically to Kafka. I'd thus, like to move it to a module that is outside of the* flink-connect

Re: Recommended module for connector ratelimiting?

Since we want to be able to use the rate limiter for multiple connectors and potentially elsewhere, plus it is user facing, this seems to be the right area. Perhaps org.apache.flink.api.common.io specifically? On Mon, Feb 25, 2019 at 5:40 PM Lakshmi Gururaja Rao wrote: > Hi all, > > The PR that

Re: Recommended module for connector ratelimiting?

Sorry, not looked into it much... but it occurred to me that it would be great to have it as "Throttling: Operator" that can be applied anywhere after a source and before a sink.  Each parallel instance of it can operate at the specified rate limit  divided by the number of parallel instances o

Re: Recommended module for connector ratelimiting?

@Thomas Weise that seems reasonable to me. I'll create a JIRA to track this. @Roshan The idea of a ratelimiting operator did come up before . The main concerns around that app

Re: Recommended module for connector ratelimiting?

Lakshmi, I would prefer we move the ratelimiter interface to the appropriate location before we merge the PR. Thanks, Thomas On Thu, Feb 28, 2019 at 5:29 PM Lakshmi Gururaja Rao wrote: > @Thomas Weise that seems reasonable to me. I'll > create a JIRA to track this. > > @Roshan > The idea of