--> On Wed, Apr 18, 2018 at 3:47 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote:
> Hi Thomas, > > I see. If exposing access for these internal classes is a must to enable > further contributions, then I would agree to do so. > I think in the future, we should also keep a closer eye on parts of the > connector code which is highly subject to modifications on a > per-environment basis and keep flexibility in mind as the base assumption > (as you stated very well, there is no "perfect" implementation for a > connector, even with best default implementations). > > Some comments on the specific issues that you mentioned: > > * use ListShards for discovery (plus also considered to limit the discovery > > to a single subtask and share the results between subtasks, which is > almost > > certainly not something I would propose to add to Flink due to additional > > deployment dependencies). > > > I think this one is most likely a direct improvement to the connector > already (minus the inter-subtask coordination). > The shard discovery method does not use other information from the > `desribeStreams` call, so the alternate API should be a direct replacement. > Yes, and Kailash is going to open a PR for that soon after a related kink on the AWS side has been sorted out: https://issues.apache.org/jira/browse/FLINK-8944 > > * ability to configure the AWS HTTP client when defaults turn out > > unsuitable for the use case. This is a very basic requirement and it is > > rather surprising that the Flink Kinesis consumer wasn't written to > provide > > access to the settings that the AWS SDK provides. > > > I see you have opened a separate JIRA for this (FLINK-9188). And yes, IMO > this is definitely something very desirable in the future. > The JIRA has a commit linked to it that shows how it will look like. I will rebase that and open a PR. > Finally, I'm also curious how much appetite for contributions in the > > connector areas there is? I see that we have now accumulated 340 open > PRs, > > and review bandwidth seems hard to come by. > > > > I would personally very like to see these contributions / improvements > happening. > In the past, the community has indeed stalled a bit in keeping up to pace > with all the contributions, but this is something that most of the > committers should have in mind and fix soon. > In the past I looked mostly at connector contributions, and would like to > get up to speed with that again shortly. > That's good to know and thanks for your help with the review! Thomas
