Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration
Thanks for the question. I tried to find a way that would work using the existing framework. Specifically, I looked into using the annotation that is partially implemented in the connector code base. That annotation relies on the getSecret method. However, I don’t see a way to make this work for wrapped connectors, like the Kafka Connect Adapter mentioned in the PIP. In that case, the wrapped connector will have arbitrary secrets that we cannot know at compile time. As such, I think we need a framework change that gives users arbitrarily map in secrets at runtime. Thanks, Michael On Fri, Jul 28, 2023 at 7:42 PM Neng Lu wrote: > Hi Michael, > > Thanks for writing the PIP for the connector secret issue. > > One question I have is why not reusing the `context.getSecret()` method > inside connectors to get sensitive values. > > In this way, no API/framework changes are needed and all we need to do is > update each connector to get the secret value with `context.getSecret()` > first. If nothing provided, then fall back to the plain text way. > > What do you think? > > On 2023/07/28 21:59:57 Michael Marshall wrote: > > Hi Pulsar Community, > > > > This is the discussion thread for PIP > > https://github.com/apache/pulsar/pull/20903. > > > > This PIP will help improve Pulsar Connector Security by giving users > > the ability to remove all plaintext secrets from their configurations. > > > > Thanks, > > Michael > > >
Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration
Hi Michael, Thanks for writing the PIP for the connector secret issue. One question I have is why not reusing the `context.getSecret()` method inside connectors to get sensitive values. In this way, no API/framework changes are needed and all we need to do is update each connector to get the secret value with `context.getSecret()` first. If nothing provided, then fall back to the plain text way. What do you think? On 2023/07/28 21:59:57 Michael Marshall wrote: > Hi Pulsar Community, > > This is the discussion thread for PIP > https://github.com/apache/pulsar/pull/20903. > > This PIP will help improve Pulsar Connector Security by giving users > the ability to remove all plaintext secrets from their configurations. > > Thanks, > Michael >
Re: [DISCUSS] PIP 289: Secure Pulsar Connector Configuration
Following up, my primary open questions are: * Should we deprecate the old way of injecting secrets? It wasn't widely used, and it does not work in all cases. (See the PIP for the old mechanism.) * Should we enable environment variable interpolation by default? It carries some risk, but in a Kubernetes environment, that risk is limited. Thanks, Michael On Fri, Jul 28, 2023 at 4:59 PM Michael Marshall wrote: > > Hi Pulsar Community, > > This is the discussion thread for PIP > https://github.com/apache/pulsar/pull/20903. > > This PIP will help improve Pulsar Connector Security by giving users > the ability to remove all plaintext secrets from their configurations. > > Thanks, > Michael
[DISCUSS] PIP 289: Secure Pulsar Connector Configuration
Hi Pulsar Community, This is the discussion thread for PIP https://github.com/apache/pulsar/pull/20903. This PIP will help improve Pulsar Connector Security by giving users the ability to remove all plaintext secrets from their configurations. Thanks, Michael
Re: [DISCUSS] Generic Secret Injection for Sink/Source Configuration
* The PIP is https://github.com/apache/pulsar/pull/20903 - Michael On Fri, Jul 28, 2023 at 4:56 PM Michael Marshall wrote: > > After looking a bit closer at the requirements for this feature, I > determined the above solution does not work because it won't work for > nested configurations. > > My new solution required a PIP: https://github.com/apache/pulsar/issues/20862. > > I'll start a new discussion thread. > > Thanks, > Michael > > On Mon, Jul 24, 2023 at 5:33 PM Michael Marshall wrote: > > > > Hi Pulsar Community, > > > > I would like to find a generic way to inject secrets into connector > > configuration without needing to rewrite existing connectors. I > > created this issue [0] to describe the current state of connector > > secret management and to discuss potential solutions. I created [1] to > > show my preferred solution. > > > > I don't think this change requires a PIP, but since it introduces a > > new configuration, I am mentioning it on the mailing list. > > > > Thanks, > > Michael > > > > [0] https://github.com/apache/pulsar/issues/20862 > > [1] https://github.com/apache/pulsar/issues/20863
Re: [DISCUSS] Generic Secret Injection for Sink/Source Configuration
After looking a bit closer at the requirements for this feature, I determined the above solution does not work because it won't work for nested configurations. My new solution required a PIP: https://github.com/apache/pulsar/issues/20862. I'll start a new discussion thread. Thanks, Michael On Mon, Jul 24, 2023 at 5:33 PM Michael Marshall wrote: > > Hi Pulsar Community, > > I would like to find a generic way to inject secrets into connector > configuration without needing to rewrite existing connectors. I > created this issue [0] to describe the current state of connector > secret management and to discuss potential solutions. I created [1] to > show my preferred solution. > > I don't think this change requires a PIP, but since it introduces a > new configuration, I am mentioning it on the mailing list. > > Thanks, > Michael > > [0] https://github.com/apache/pulsar/issues/20862 > [1] https://github.com/apache/pulsar/issues/20863
Re: Upgrade to 2.11.2 fail
When I commented out # supportedNamespaceBundleSplitAlgorithms=range_equally_divide,topic_count_equally_divide,specified_positions_divide,flow_or_qps_equally_divide # defaultNamespaceBundleSplitAlgorithm=range_equally_divide and I presume it relies on defaults in code, then it at least starts. I also suggest that the Exception puts the brokerConfig value in the message as well as the supported algos (Line 179 in PulsarBrokerStarter.java), so there is a chance to figure out what is going wrong here. // Niclas On 2023-07-28 11:59, Niclas Hedhman wrote: Hi, I am trying to upgrade from 2.10.3 to 2.11.2 and I am getting the following right at start-up and I don't understand the message at all. 2023-07-28T09:46:07,973Z [jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27] error Uncaught exception in thread main: The given supported namespace bundle split algorithm has unavailable algorithm. Available algorithms are [range_equally_divide, topic_count_equally_divide, specified_positions_divide] java.lang.IllegalArgumentException: The given supported namespace bundle split algorithm has unavailable algorithm. Available algorithms are [range_equally_divide, topic_count_equally_divide, specified_positions_divide] at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.(PulsarBrokerStarter.java:179) at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331) I don't understand what a "namespace bundle split" is, but I found the following in my broker.conf; # Supported algorithms name for namespace bundle split. # "range_equally_divide" divides the bundle into two parts with the same hash range size. # "topic_count_equally_divide" divides the bundle into two parts with the same topics count. # "specified_positions_divide" divides the bundle into several parts by the specified positions. supportedNamespaceBundleSplitAlgorithms=range_equally_divide,topic_count_equally_divide,specified_positions_divide,flow_or_qps_equally_divide # Default algorithm name for namespace bundle split defaultNamespaceBundleSplitAlgorithm=range_equally_divide And since default is set to an acceptable value, I presume that my namespace has gotten it set to something else somehow, probably on creation (from Java client library) in a previous version (possibly 2.10.0). Is this a new feature? And how am I supposed to make the upgrade? Is there any upgrade/migration documentation available? Cheers Niclas
Upgrade to 2.11.2 fail
Hi, I am trying to upgrade from 2.10.3 to 2.11.2 and I am getting the following right at start-up and I don't understand the message at all. 2023-07-28T09:46:07,973Z [jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27] error Uncaught exception in thread main: The given supported namespace bundle split algorithm has unavailable algorithm. Available algorithms are [range_equally_divide, topic_count_equally_divide, specified_positions_divide] java.lang.IllegalArgumentException: The given supported namespace bundle split algorithm has unavailable algorithm. Available algorithms are [range_equally_divide, topic_count_equally_divide, specified_positions_divide] at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.(PulsarBrokerStarter.java:179) at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331) I don't understand what a "namespace bundle split" is, but I found the following in my broker.conf; # Supported algorithms name for namespace bundle split. # "range_equally_divide" divides the bundle into two parts with the same hash range size. # "topic_count_equally_divide" divides the bundle into two parts with the same topics count. # "specified_positions_divide" divides the bundle into several parts by the specified positions. supportedNamespaceBundleSplitAlgorithms=range_equally_divide,topic_count_equally_divide,specified_positions_divide,flow_or_qps_equally_divide # Default algorithm name for namespace bundle split defaultNamespaceBundleSplitAlgorithm=range_equally_divide And since default is set to an acceptable value, I presume that my namespace has gotten it set to something else somehow, probably on creation (from Java client library) in a previous version (possibly 2.10.0). Is this a new feature? And how am I supposed to make the upgrade? Is there any upgrade/migration documentation available? Cheers Niclas
