Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration

2023-07-28 Thread Michael Marshall
Thanks for the question. I tried to find a way that would work using the existing framework. Specifically, I looked into using the annotation that is partially implemented in the connector code base. That annotation relies on the getSecret method. However, I don’t see a way to make this work for w

Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration

2023-07-28 Thread Neng Lu
Hi Michael, Thanks for writing the PIP for the connector secret issue. One question I have is why not reusing the `context.getSecret()` method inside connectors to get sensitive values. In this way, no API/framework changes are needed and all we need to do is update each connector to get the

Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration

2023-07-28 Thread Michael Marshall
Following up, my primary open questions are: * Should we deprecate the old way of injecting secrets? It wasn't widely used, and it does not work in all cases. (See the PIP for the old mechanism.) * Should we enable environment variable interpolation by default? It carries some risk, but in a Kube

[DISCUSS] PIP 289: Secure Pulsar Connector Configuration

2023-07-28 Thread Michael Marshall
Hi Pulsar Community, This is the discussion thread for PIP https://github.com/apache/pulsar/pull/20903. This PIP will help improve Pulsar Connector Security by giving users the ability to remove all plaintext secrets from their configurations. Thanks, Michael

Re: [DISCUSS] Generic Secret Injection for Sink/Source Configuration

2023-07-28 Thread Michael Marshall
* The PIP is https://github.com/apache/pulsar/pull/20903 - Michael On Fri, Jul 28, 2023 at 4:56 PM Michael Marshall wrote: > > After looking a bit closer at the requirements for this feature, I > determined the above solution does not work because it won't work for > nested configurations. > > M

Re: [DISCUSS] Generic Secret Injection for Sink/Source Configuration

2023-07-28 Thread Michael Marshall
After looking a bit closer at the requirements for this feature, I determined the above solution does not work because it won't work for nested configurations. My new solution required a PIP: https://github.com/apache/pulsar/issues/20862. I'll start a new discussion thread. Thanks, Michael On M

Re: Upgrade to 2.11.2 fail

2023-07-28 Thread Niclas Hedhman
When I commented out # supportedNamespaceBundleSplitAlgorithms=range_equally_divide,topic_count_equally_divide,specified_positions_divide,flow_or_qps_equally_divide # defaultNamespaceBundleSplitAlgorithm=range_equally_divide and I presume it relies on defaults in code, then it at least sta

Upgrade to 2.11.2 fail

2023-07-28 Thread Niclas Hedhman
Hi, I am trying to upgrade from 2.10.3 to 2.11.2 and I am getting the following right at start-up and I don't understand the message at all. 2023-07-28T09:46:07,973Z [jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27] error Uncaught exception in thread main: The given supported names