mosche commented on issue #23206:
URL: https://github.com/apache/beam/issues/23206#issuecomment-1271636243
> Isn't it a responsibility of users to check their configuration on workers
and not Beam/KinesisIO? I mean why KinesisIO (or any other AWS IO connector)
has to check if it's matching or not?
Beam's responsibility is backwards compatibility in this case,
@aromanenko-dev.
If we add support for custom profile names for `ProfileCredentialsProvider`
(will say we extract/serialize the profile name in the respective pipeline
option), it gets a bit tricky as I tried to explain above. When serializing a
provider that was set programmatically, we cannot distinguish if the user
provided the profile name or if it was loaded from the system environment (e.g.
`AWS_PROFILE=master`). If the latter is the case and the system environment on
a worker node, where the credentials are eventually used, looks differently
(e.g. `AWS_PROFILE=worker`) things can easily break for pipelines that used to
run successfully before: we would force the `master` profile on a worker even
though it should use the `worker` profile there 💥
On the other hand, in case we should decide to not support this, it's still
important to check the system environment during serialization of pipeline
options. It's the only way to detect if the user set an unsupported
ProfileCredentialsProvider programmatically with a custom profile name
(`ProfileCredentialsProvider.create("myprofile")`). If we don't check it and
silently ignore the configuration (as done now), things break fairly randomly
on workers as experienced here in this ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
