Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration

2023-08-11 Thread Baodi Shi
Hi, Sorry for the late reply. I think we can avoid adding a new config: interpolate_secrets_into_config_map. https://github.com/apache/pulsar/pull/20903#issuecomment-1674570878 Thanks, Baodi Shi On Aug 11, 2023 at 00:34:05, Michael Marshall wrote: > * Should we deprecate the old way of inje

Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration

2023-08-10 Thread Michael Marshall
> * Should we deprecate the old way of injecting secrets? It wasn't > widely used, and it does not work in all cases. (See the PIP for the > old mechanism.) I propose we leave the old way of injecting secrets in place. This will prevent unnecessary breaking changes. I'll add some new docs to make

Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration

2023-07-28 Thread Michael Marshall
Thanks for the question. I tried to find a way that would work using the existing framework. Specifically, I looked into using the annotation that is partially implemented in the connector code base. That annotation relies on the getSecret method. However, I don’t see a way to make this work for w

Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration

2023-07-28 Thread Neng Lu
Hi Michael, Thanks for writing the PIP for the connector secret issue. One question I have is why not reusing the `context.getSecret()` method inside connectors to get sensitive values. In this way, no API/framework changes are needed and all we need to do is update each connector to get the

Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration

2023-07-28 Thread Michael Marshall
Following up, my primary open questions are: * Should we deprecate the old way of injecting secrets? It wasn't widely used, and it does not work in all cases. (See the PIP for the old mechanism.) * Should we enable environment variable interpolation by default? It carries some risk, but in a Kube

[DISCUSS] PIP 289: Secure Pulsar Connector Configuration

2023-07-28 Thread Michael Marshall
Hi Pulsar Community, This is the discussion thread for PIP https://github.com/apache/pulsar/pull/20903. This PIP will help improve Pulsar Connector Security by giving users the ability to remove all plaintext secrets from their configurations. Thanks, Michael