*Hi Harsha/Maulin,I am following KIP-486 and KIP-492 and it seems https://github.com/apache/kafka/pull/7090 <https://github.com/apache/kafka/pull/7090> is the right solution when one wants to register custom factory class for KeyManager and TrustManager. User can easily configure custom implementation of TrustManager and KeyManager using factory Provider class.Configuration of the provider is also simple and straightforward: 1. Create custom provider which defines factory classes for KeyManager and TrustManagerpublic class CustomProvider extends Provider { public CustomProvider() { super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore"); super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory"); super.put("TrustManagerFactory.CUSTOM","customTrustManagerFactory"); }} 1. Register provider at broker startupjava.security.Security.addProvider(new CustomProvider());However, this approach is useful when one wants to implement custom TrustManager for X509 certs by extending X509ExtendedKeyManager and implement various abstract methods such as: checkClientTrusted, checkServerTrusted, etc..In JDK, default implementation class of X509ExtendedKeyManager is X509TrustManagerImpl and one can’t extend or delegate calls to this class because it’s final and same is applicable for other available providers such as : BouncyCastleProviderTurstManager/KeyManager mainly serve two purposes: 1. Provide certs/key2. Perform validationX509TrustManagerImpl performs various RFC specified validations.#7090 can be helpful when user has both above asks. However, problem defined at KIP-486 has different ask where user wants to provide certs/key without copying/implementing Manager class because all the available Manager classes are final and can’t be extended or delegated. And security team in most of the companies don’t allow custom/copying provider in order to get up to date with various RFC validations provided into standard jdk provider.Many users manage keys and certs into KMS and sometimes it’s not feasible to copy them to file-system instead directly use them from the memory. So, KIP-486 provides a custom way to load keys/certs without implementing security-provider.Thanks,Rajan*
On Wed, Aug 28, 2019 at 2:18 PM Maulin Vasavada <maulin.vasav...@gmail.com> wrote: > > > Hi Harsha > > As we synced-up offline on this topic, we hope you don't have any more > clarifications that you are seeking. If that is the case, can you please > help us move this forward and discuss what changes you would expect on the > KIP design in order to make it valuable contribution? > > Just FYI - we verified our primary design change with the author of Sun's > X509 Trustmanager implementation and the outcome is that what we are > proposing makes sense at the heart of it - "Instead of writing TrustManager > just plugin the Trust store". We are open to discuss additional changes > that you/anybody else would like to see on the functionality however. > > > Thanks > Maulin > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <maulin.vasav...@gmail.com> > wrote: > >> Hi Harsha >> >> Any response on my question? I feel this KIP is worth accommodating. Your >> help is much appreciated. >> >> Thanks >> Maulin >> >> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada < >> maulin.vasav...@gmail.com> wrote: >> >>> Hi Harsha >>> >>> I've examined the SPIFFE provider more and have one question - >>> >>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below >>> location, would you really still write the Provider? *OR* >>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass >>> the KeyStore from set of certs returned by spiffeIdManager. >>> getTrustedCerts()? >>> >>> >>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100 >>> >>> >>> /** >>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID >>>> configured in >>>> * the java.security property ssl.spiffe.accept >>>> * >>>> * If the authorized spiffe ids list is empty any spiffe id is authorized >>>> * >>>> * @param chain an array of X509Certificate that contains the Peer's >>>> SVID to be validated >>>> * @throws CertificateException when either the certificates doesn't >>>> have a SPIFFE ID or the SPIFFE ID is not authorized >>>> */ >>>> static void checkSpiffeId(X509Certificate[] chain) throws >>>> CertificateException { >>> >>> >>> >>> Thanks >>> Maulin >>> >>> >>> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io> >>> wrote: >>> >>>> Maulin, >>>> The code parts you are pointing are specific for Spiffe >>>> and if >>>> you are talking about validate method which uses PKIX check like any >>>> other provider does. >>>> If you want to default to SunJSSE everywhere you can do so by delegating >>>> the calls in these methods to SunJSSE provider. >>>> >>>> TrustManagerFactory tmf = TrustManagerFactory >>>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use >>>> tmf.chekServerTrusted() >>>> or use >>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if >>>> you want a specific provider. >>>> >>>> -Harsha >>>> >>>> >>>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada < >>>> maulin.vasav...@gmail.com> >>>> wrote: >>>> >>>> > Okay, so I take that you guys agree that I have to write a 'custom' >>>> > algorithm and a provider to make it work , correct? >>>> > >>>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an >>>> > implementation per say , ..." , I diagree. You can refer to https:// >>>> > >>>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/ >>>> > SpiffeTrustManager.java#L91 and >>>> > >>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >>>> > provider/CertificateUtils.java#L100 >>>> > >>>> > "that code" is the customization you have for the custom way to check >>>> > something on top of regular checks. That method is NOT doing custom >>>> > truststore loading. It is validating/verifying something in the >>>> "custom" >>>> > way with spiffeId. >>>> > I bet that without that you won't have a need of the custom algorithm >>>> in >>>> > the first place. >>>> > >>>> > Let me know if you agree to this. >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mpr...@gmail.com> >>>> wrote: >>>> > >>>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use >>>> the >>>> > KIP492, you need to create a new provider, register a *new* custom >>>> > algorithm for your keymanager and trustmanager factory >>>> implementations. >>>> > After this, the kafka server configuration can be done as given below >>>> > >>>> > # Register the provider class with custom algorithm, say CUSTOM >>>> security. >>>> > provider.classes=com.company.security.CustomProvider >>>> > <http://security.provider.classes >>>> =com.company.security.customprovider/> >>>> > >>>> > # Register the keymanager and trustmanager algorithms >>>> > # These algorithms indicate that the Keymanager and Trustmanagers >>>> > registered under the algorithm "CUSTOM" needs to be used >>>> > ssl.trustmanager.algorithm=CUSTOM >>>> > ssl.keymanager.algorithm=CUSTOM >>>> > >>>> > And the customprovider looks like this... >>>> > >>>> > public class CustomProvider extends Provider { >>>> > public CustomProvider() { >>>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore"); >>>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory"); >>>> > super.put("TrustManagerFactory.CUSTOM", >>>> > "customTrustManagerFactory"); >>>> > } >>>> > } >>>> > >>>> > The PR for this is in review and can be found here: >>>> https://github.com/ >>>> > apache/kafka/pull/7090 >>>> > This PR includes the fixed insertProviderAt function call. >>>> > >>>> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io> >>>> > wrote: >>>> > >>>> > Answers are in-line >>>> > >>>> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada < >>>> maulin.vasavada@gmail. >>>> > com >>>> > >>>> > wrote: >>>> > >>>> > Hi Colin >>>> > >>>> > When I refer to "standard" or "custom" algorithms I am following Java >>>> > security Provider Terminology. You can refer to >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ >>>> > StandardNames.html#TrustManagerFactory link I provided earlier in the >>>> > emails. It says PKIX is the default Algorithm for TrustManagerFactory. >>>> > >>>> > 1. For SPIFFE, I am not sure why you are saying 'it does not implement >>>> > custom algorithms' because the following file clearly indicates that >>>> it >>>> > does use custom algorithm- >>>> > >>>> > >>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >>>> > >>>> > provider/SpiffeProvider.java#L17 >>>> > >>>> > Algorithm value: >>>> > >>>> > >>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >>>> > >>>> > provider/SpiffeProviderConstants.java#L6 >>>> > >>>> > @Harsha do you want to chime in since you use that provider? >>>> > >>>> > Here the "Custom" Algorithm is not an implementation per say , rather >>>> > >>>> > used >>>> > >>>> > to invoke the custom trust store factory and key manager factory. You >>>> are >>>> > not moving away from "standard" alogrithms that are available. >>>> > >>>> > >>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >>>> > provider/SpiffeTrustManager.java >>>> > >>>> > As you can see it delegates all the calls of verification of >>>> certificate >>>> > >>>> > to >>>> > >>>> > the default implementation available. >>>> > So in our implementation we still use PKIX to verify the certificate >>>> > chain. So you are not losing anything here and Spiffe is not >>>> reimplementing >>>> > the verification process. >>>> > >>>> > 2. I already mentioned in my 3rd point, in my previous post, why using >>>> > >>>> > ssl.provider does NOT work. I updated KIP-486 in "rejected >>>> > >>>> > alternatives" >>>> > >>>> > also why ssl.provider does not work. >>>> > >>>> > As mentioned before , provider is the right way to go. I am still not >>>> > understanding the gap is. >>>> > If I understand correctly your argument is , provider is going to ask >>>> to >>>> > implement a custom algorithm. >>>> > My answer to that is , no you are not re-implementing the algorithm. >>>> > >>>> > Please >>>> > >>>> > check the above link , TrustManager implementation it delegates the >>>> calls >>>> > back. There is no need to implement your own here. >>>> > >>>> > 3. Security.insertProviderAt() comments were based on assumption if >>>> > >>>> > KIP-492 >>>> > >>>> > changes are done and we use that mechanism to configure providers >>>> > >>>> > instead >>>> > >>>> > of ssl.provider configuration. >>>> > >>>> > KIP-492 has patch available and going through review. >>>> > >>>> > Can you read my all the points, I mentioned in my previous post, very >>>> > >>>> > carefully? I am covering all the aspects in explaining. I am open to >>>> > >>>> > still >>>> > >>>> > discuss more to clarify any doubts. >>>> > >>>> > "3. If we just use existing ssl.provider kafka configuration then our >>>> > provider will be used in SSLContext.getInstance(protocol, provider) >>>> call >>>> > >>>> > in >>>> > >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> >>>> and >>>> > if our provider does not have >>>> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we >>>> > >>>> > tested >>>> > >>>> > it). Example: In MyProvider sample above you see that I didn't add >>>> > SSLContext.TLSv1 as >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider >>>> you >>>> > don't have this challenge since you are planning to bypass >>>> ssl.provider >>>> > >>>> > as >>>> > >>>> > you mention in the KIP-492." >>>> > >>>> > Yes here you need to pass the protocol that your >>>> KeyManager/TrustManager >>>> > registered with and in no way its deviating from TLS RFC spec. >>>> > >>>> > >>>> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/ >>>> > javax/net/ssl/SSLContext.java#L134 >>>> > >>>> > My suggestion here is for you to implement a simple Security Provider >>>> as >>>> > you did before and register a Algorithm. You can use the existing >>>> > implementation in SpiffeProvider and plug in changes where you need to >>>> > retrieve the certificates from by making RPC call. >>>> > >>>> > Run an end-to-end test with Kafka broker coming up with your provider >>>> > making calls to RPC call. You do need to pass the "custom algorithm" >>>> that >>>> > you registered in your key manager to invoke the provider. I think >>>> your >>>> > concern is this approach is replacing the existing known ciphersuites >>>> and >>>> > certificate validation provided by java. Which its not. >>>> > >>>> > Now test the TLS connection you can do so via openssl -s_client >>>> options >>>> > >>>> > to >>>> > >>>> > test if everything else is working. >>>> > >>>> > I am happy to share configs that we used if you are interested. >>>> > >>>> > Thanks, >>>> > Harsha >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cmcc...@apache.org> >>>> > >>>> > wrote: >>>> > >>>> > Hi Maulin, >>>> > >>>> > A lot of JSSE providers don't implement custom algorithms. Spire is a >>>> > >>>> > good >>>> > >>>> > example of a JSSE provider that doesn't, and yet is still useful to >>>> > >>>> > many >>>> > >>>> > people. Your JSSE provider can work fine even if it doesn't implement >>>> a >>>> > custom algorithm. >>>> > >>>> > Maybe I'm missing something, but I don't understand the discussion of >>>> > Security.insertProviderAt() that you included. SslEngineBuilder >>>> doesn't >>>> > >>>> > use >>>> > >>>> > that API to get the security provider. Instead, it calls >>>> > "SSLContext.getInstance(protocol, provider)", where provider is the >>>> > >>>> > name >>>> > >>>> > of the provider. >>>> > >>>> > best, >>>> > Colin >>>> > >>>> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote: >>>> > >>>> > On top of everything above I feel strongly to add the 4th point which >>>> > >>>> > is >>>> > >>>> > based on Java APIs for TrustManagerFactory.init(KeyStore) ( >>>> > >>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/ >>>> > TrustManagerFactory.html#init(java.security.KeyStore >>>> > <http://java.security.keystore/>) >>>> > ) >>>> > >>>> > and KeyManagerFactory.init(KeyStore, char[]) ( >>>> > >>>> > >>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory >>>> . >>>> > >>>> > >>>> > html#init(java.security.KeyStore <http://java.security.keystore/ >>>> >,%20char[]) >>>> > >>>> > >>>> > ). >>>> > >>>> > 4. The above APIs are intended to support providing "trust/key >>>> > >>>> > material" >>>> > >>>> > from the user without having to write their own >>>> > >>>> > TrustManager/KeyManagers. >>>> > >>>> > To quote from the TrustManagerFactory.init()'s documentation >>>> > >>>> > "Initializes >>>> > >>>> > this factory with a source of certificate authorities and related >>>> trust >>>> > material." >>>> > To quote from the KeyManagerFactory.init()'s documentation >>>> "Initializes >>>> > this factory with a source of key material." >>>> > >>>> > Based on this it is clear that there is a flexibility provided by Java >>>> > >>>> > to >>>> > >>>> > to enable developers to provide the required trust/key material loaded >>>> > >>>> > from >>>> > >>>> > "anywhere" without requiring them to write custom provider OR >>>> trust/key >>>> > managers. This same flexibility is reflected in Kafka code also where >>>> > >>>> > it >>>> > >>>> > loads the trust/keys from a local file and doesn't require writing a >>>> > Provider necessarily. If we do NOT have a custom algorithm, it makes >>>> > >>>> > less >>>> > >>>> > sense to write a Provider. >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada < >>>> > >>>> > maulin.vasav...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Hi Harsha/Colin >>>> > >>>> > I did the sample with a custom Provider for TrustStoreManager and >>>> tried >>>> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by >>>> > adding Provider programmatically instead of relying on >>>> > >>>> > ssl.provider+java. >>>> > >>>> > security. The below sample is followed by my detailed findings. I'll >>>> > appreciate if you can go through it carefully and see >>>> > >>>> > if you >>>> > >>>> > see my point. >>>> > >>>> > package providertest; >>>> > >>>> > import java.security.Provider <http://java.security.provider/> >>>> <http:// >>>> > java.security.provider/>; >>>> > >>>> > public class MyProvider extends Provider { >>>> > >>>> > private static final String name = "MyProvider"; private static double >>>> > version = 1.0d; >>>> > private static String info = "Maulin's SSL Provider v"+version; >>>> > >>>> > public MyProvider() { >>>> > super(name, version, info); >>>> > this.put("TrustManagerFactory.PKIX", >>>> > >>>> > "providertest.MyTrustManagerFactory"); >>>> > >>>> > } >>>> > } >>>> > >>>> > *Details:* >>>> > >>>> > KIP-492 documents that it will use Security.addProvider() assuming it >>>> > >>>> > will >>>> > >>>> > add it as position '0' which is not a correct assumption. The >>>> > addProvider()'s documentation says it will add it to the last >>>> available >>>> > position. You may want to correct that to say >>>> > Security.insertProviderAt(provider, 1). >>>> > >>>> > Now coming back to our specific discussion, >>>> > >>>> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add >>>> > >>>> > that >>>> > >>>> > provider in the provider list via Security.addProvider() the position >>>> > >>>> > where >>>> > >>>> > it gets added doesn't matter (even if you don't end up adding it as >>>> > >>>> > first >>>> > >>>> > entry) since that is the ONLY provider for SPIFFE specific algorithm >>>> > >>>> > you >>>> > >>>> > might have. >>>> > >>>> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which >>>> > >>>> > means >>>> > >>>> > we have to use X509, PKIX etc "Standard Algorithms" (( >>>> > >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ >>>> > StandardNames.html >>>> > )) >>>> > >>>> > in our provider to override the TrustStoreManager (see my sample code) >>>> > >>>> > and >>>> > >>>> > KeyStoreManger and KeyManager. This creates another challenge >>>> > >>>> > mentioned in >>>> > >>>> > the below point. >>>> > >>>> > 2. In order to make our Provider for loading custom TrustStore work, >>>> we >>>> > have to add the provider as 'first' in the list since there are others >>>> > >>>> > with >>>> > >>>> > the same algorithm. >>>> > >>>> > However, the programatic way of adding provider >>>> > (Security.insertProviderAt()) is *not* deterministic for ordering >>>> since >>>> > different code can call the method for a different provider and >>>> > >>>> > depending >>>> > >>>> > upon the order of the call our provider can be first or pushed down >>>> the >>>> > list. This can happen very well in any client application using Kafka. >>>> > >>>> > This >>>> > >>>> > is specially problematic for a case when you want to guarantee order >>>> > >>>> > for a >>>> > >>>> > Provider having "Standard Algorithms". >>>> > >>>> > If we add our provider in java.security file that definitely >>>> guarantees >>>> > the order(unless somebody calls removeProvider() which is less >>>> > >>>> > likely). But >>>> > >>>> > if we add our provider in java.security file it will defeat the >>>> > >>>> > purpose of >>>> > >>>> > the KIP-492. >>>> > >>>> > In the gist - Apache Kafka must not rely on "un-deterministic" method >>>> > >>>> > to >>>> > >>>> > rely on Provider ordering. >>>> > >>>> > 3. If we just use existing ssl.provider kafka configuration then our >>>> > provider will be used in SSLContext.getInstance(protocol, provider) >>>> > >>>> > call in >>>> > >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> >>>> and >>>> > if our provider does not have implementation for >>>> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks >>>> > >>>> > (we >>>> > >>>> > tested it). Example: >>>> > >>>> > In >>>> > >>>> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider >>>> you >>>> > don't have this challenge since you are planning to bypass >>>> > >>>> > ssl.provider as >>>> > >>>> > you mention in the KIP-492. >>>> > >>>> > *Overall summary,* >>>> > >>>> > 1. Any provider based mechanisms- a) existing ssl.provider and >>>> > >>>> > b)KIP-492, >>>> > >>>> > for loading key/trust store using "Standard Algorithms" do not work >>>> > >>>> > 2. Approach suggested in our KIP-486 works without any issue and it is >>>> > *not* our context specific solve >>>> > >>>> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary >>>> changes >>>> > and not contradicting or redundent. >>>> > >>>> > If you want we can do a joint session somehow to walk through the >>>> > >>>> > sample I >>>> > >>>> > have and various experiments I did. I would encourage you to do >>>> similar >>>> > exercise by writing a Provider for "Standard Algorithm" for >>>> > TrustStoreManager (like our needs) and see what you find since only >>>> > >>>> > writing >>>> > >>>> > samples can bring out the complexity/challenges we face. >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada < >>>> > >>>> > maulin.vasavada@gmail. >>>> > >>>> > com> wrote: >>>> > >>>> > Just to update - still working on it. Get to work only on and off on >>>> > >>>> > it :( >>>> > >>>> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada < >>>> > >>>> > maulin.vasav...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Hi Harsha >>>> > >>>> > Let me try to write samples and will let you know. >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <harsha...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Hi Maulin, >>>> > With java security providers can be as custom you would >>>> > >>>> > like >>>> > >>>> > it to >>>> > be. If you only want to to implement a custom way of loading the >>>> > >>>> > keystore >>>> > >>>> > and truststore and not implement any protocol/encryption handling you >>>> can >>>> > leave them empty and no need to implement. Have you looked into the >>>> links I >>>> > pasted before? >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > >>>> > >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore. >>>> > >>>> > java >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/ >>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/> >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > >>>> > >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. >>>> > >>>> > java >>>> > >>>> > Can you please tell me which methods are too complex in above to >>>> > >>>> > implement >>>> > >>>> > or unnecessary? You are changing anything in SSL/TLS implementations >>>> > provided by >>>> > >>>> > All of the implementations delegating the checks to the default >>>> > implementation anyway. >>>> > Spire agent is an example, its nothing but a GRPC server listening >>>> > >>>> > on a >>>> > >>>> > unix domain socket . Above code is making a RPC call to the local >>>> > >>>> > daemon >>>> > >>>> > to >>>> > get the certificate and keys. The mechanics are pretty much same as >>>> > >>>> > what >>>> > >>>> > you are asking for. >>>> > >>>> > Thanks, >>>> > Harsha >>>> > >>>> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada < >>>> > >>>> > maulin.vasav...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Imagine a scenario like - We know we have a custom KMS and as a >>>> > >>>> > Kafka >>>> > >>>> > owner >>>> > >>>> > we want to comply to using that KMS source to load keys/certs. As >>>> > >>>> > a >>>> > >>>> > Kafka >>>> > >>>> > owner we know how to integrate with KMS but doesn't necessarily >>>> > >>>> > have >>>> > >>>> > to >>>> > >>>> > know anything about cipher suites, algorithms, and SSL/TLS >>>> > >>>> > implementation. >>>> > >>>> > Going the Provider way requires to know lot more than we should, >>>> > >>>> > isn't it? >>>> > >>>> > Not that we would have concern/shy-away knowing those details - >>>> > >>>> > but >>>> > >>>> > if we >>>> > >>>> > don't have to - why should we? >>>> > >>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada < >>>> > >>>> > maulin.vasav...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Hi Harsha >>>> > >>>> > We don't have spire (or similar) agents and we do not have >>>> > >>>> > keys/certs >>>> > >>>> > locally on any brokers. To elaborate more on my previous email, >>>> > >>>> > I agree that Java security Providers are used in much broader >>>> > >>>> > sense >>>> > >>>> > - to >>>> > >>>> > have a particular implementation of an algorithm, use specific >>>> > >>>> > cipher >>>> > >>>> > suites for SSL , OR in our current team's case have a >>>> > >>>> > particular >>>> > >>>> > way to >>>> > >>>> > leverage pre-generated SSL sessions. However, the scope of our >>>> > >>>> > KIP >>>> > >>>> > (486) >>>> > >>>> > is >>>> > >>>> > much restricted than that. We merely intend to provide a custom >>>> > keystore/truststore for our SSL connections and not really worry >>>> > >>>> > about >>>> > >>>> > underlying specific SSL/TLS implementation. This simplifies it >>>> > >>>> > a >>>> > >>>> > lot for >>>> > >>>> > us to keep the concerns separate and clear. >>>> > >>>> > I feel our approach is more complimentary such that it allows >>>> > >>>> > for >>>> > >>>> > using >>>> > >>>> > keystores of choice while retaining the flexibility to use any >>>> > underlying/available Provider for actually making the SSL call. >>>> > >>>> > We agree with KIP-492's approach based on Providers (and Java's >>>> > recommendation), but also strongly believe that our approach can >>>> > >>>> > compliment >>>> > >>>> > it very effectively for reasons explained above. >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani < >>>> > >>>> > ka...@harsha.io >>>> > >>>> > wrote: >>>> > >>>> > Hi Maulin, >>>> > >>>> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada < >>>> > >>>> > maulin.vasavada@gmail. >>>> > >>>> > com> >>>> > wrote: >>>> > >>>> > Hi Harsha >>>> > >>>> > The reason we rejected the SslProvider route is that - we >>>> > >>>> > only >>>> > >>>> > needed >>>> > >>>> > a >>>> > >>>> > custom way to load keys/certs. Not touch any policy that >>>> > >>>> > existing >>>> > >>>> > Providers >>>> > >>>> > govern like SunJSSE Provider. >>>> > >>>> > We have exactly the same requirements to load certs and keys >>>> > >>>> > through >>>> > >>>> > spire >>>> > >>>> > agent. We used security.provider to do that exactly. I am not >>>> > >>>> > sure >>>> > >>>> > why >>>> > >>>> > you >>>> > >>>> > would be modifying any policies provided by default SunJSSE >>>> > >>>> > provider. >>>> > >>>> > Can >>>> > >>>> > you give me an example of having custom provider that will >>>> > >>>> > override an >>>> > >>>> > existing policy in SunJSSE provider. >>>> > >>>> > As pointed out earlier, this kip >>>> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/ >>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config >>>> > >>>> > allows >>>> > you to load security.provider through config. >>>> > Take a look at the examples I gave before >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > >>>> > >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. >>>> > >>>> > java >>>> > >>>> > It registers KeyManagerFactory and TrustManagerFactory and >>>> > >>>> > Keystore >>>> > >>>> > algorithm. >>>> > Implement your custom way of loading Keystore in here >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > >>>> > >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore. >>>> > >>>> > java >>>> > >>>> > and Trust manager like here >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/ >>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/> >>>> > >>>> > In your Kafka client you can set the security.provider to your >>>> > >>>> > custom >>>> > >>>> > implementation and with this fix >>>> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set >>>> > keyManagerAlgorigthm and trustManagerAlgorithm configs. >>>> > >>>> > All of this is in your clients and broker side and do not need >>>> > >>>> > to >>>> > >>>> > touch >>>> > >>>> > any >>>> > policy changes at JVM level. You'll register the providers in >>>> > >>>> > the >>>> > >>>> > priority >>>> > >>>> > order and can still have SunJSSE provider and have your custom >>>> > >>>> > provider >>>> > >>>> > to >>>> > >>>> > implement the key and trust managers. >>>> > >>>> > The ask here is different than KIP-492. We don't have any need >>>> > >>>> > to >>>> > >>>> > modify/specify the algorithm parameter. Does that make sense? >>>> > >>>> > The ask in KIP is introducing new interfaces where the KIP's >>>> > goal/motivation can be achieved through the security.provider >>>> > >>>> > and >>>> > >>>> > we >>>> > >>>> > worked >>>> > on similar goal without touching any Keystore or Truststore >>>> > >>>> > interfaces. >>>> > >>>> > My advise is against changing or introducing new interfaces >>>> > >>>> > when >>>> > >>>> > it can >>>> > >>>> > work through security.provider. >>>> > >>>> > Thanks, >>>> > Harsha >>>> > >>>> > Thanks >>>> > >>>> > Maulin >>>> > >>>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani < >>>> > >>>> > ka...@harsha.io> >>>> > >>>> > wrote: >>>> > >>>> > In your KIP you added security. provider as rejected >>>> > >>>> > alternative >>>> > >>>> > and >>>> > >>>> > specified "its not the correct way". Do you mind explaining >>>> > >>>> > why >>>> > >>>> > its >>>> > >>>> > not? I >>>> > >>>> > didn't find any evidence in Java docs to say so. Contrary to >>>> > >>>> > your >>>> > >>>> > statement >>>> > >>>> > it does say in the java docs >>>> > " However, please note that a provider can be used to >>>> > >>>> > implement >>>> > >>>> > any >>>> > >>>> > security service in Java that uses a pluggable architecture >>>> > >>>> > with >>>> > >>>> > a >>>> > >>>> > choice >>>> > >>>> > of implementations that fit underneath." >>>> > >>>> > Java Security Providers have been used by other projects to >>>> > >>>> > provide >>>> > >>>> > such >>>> > >>>> > integration . I am not sure if you looked into Spiffe >>>> > >>>> > project to >>>> > >>>> > efficiently distribute certificates but here is an example of >>>> > >>>> > Java >>>> > >>>> > provider >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > >>>> > >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. >>>> > >>>> > java which >>>> > obtains certificates from local daemons. >>>> > These integrations are being used in Tomcat, Jetty etc.. We >>>> > >>>> > are >>>> > >>>> > also >>>> > >>>> > using >>>> > >>>> > Security provider to do the same in our Kafka clusters. So >>>> > >>>> > unless I >>>> > >>>> > see >>>> > >>>> > more evidence why security.provider doesn't work for you >>>> > >>>> > adding >>>> > >>>> > new >>>> > >>>> > interfaces while there exists more cleaner way of achieving >>>> > >>>> > the >>>> > >>>> > goals >>>> > >>>> > of >>>> > >>>> > this KIP is unnecessary and breaks the well known security >>>> > >>>> > interfaces >>>> > >>>> > provided by Java itself. >>>> > >>>> > Thanks, >>>> > Harsha >>>> > >>>> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani < >>>> > >>>> > ka...@harsha.io >>>> > >>>> > wrote: >>>> > >>>> > Hi Maulin, >>>> > Not sure if you looked at my previous replies. This >>>> > >>>> > changes >>>> > >>>> > are not required as there is already security Provider to do >>>> > >>>> > what you >>>> > >>>> > are >>>> > >>>> > proposing. This KIP >>>> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/ >>>> > >>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config >>>> > >>>> > also >>>> > >>>> > addresses easy registration of such providers. >>>> > >>>> > Thanks, >>>> > Harsha >>>> > >>>> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada >>>> > >>>> > <maulin.vasavada@gmail. >>>> > >>>> > com> wrote: >>>> > >>>> > Bump! Can somebody please review this? >>>> > >>>> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada < >>>> > >>>> > maulin.vasav...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Bump! Can somebody please review this? >>>> > >>>> > -- >>>> > Thanks, >>>> > M.Sai Sandeep >>>> > >>>> > >>>> >>> On Wed, Aug 28, 2019 at 2:18 PM Maulin Vasavada <maulin.vasav...@gmail.com> wrote: > > > ---------- Forwarded message --------- > From: Maulin Vasavada <maulin.vasav...@gmail.com> > Date: Wed, Aug 28, 2019 at 1:51 PM > Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and > TrustStore > To: <dev@kafka.apache.org> > > > Hi Harsha > > As we synced-up offline on this topic, we hope you don't have any more > clarifications that you are seeking. If that is the case, can you please > help us move this forward and discuss what changes you would expect on the > KIP design in order to make it valuable contribution? > > Just FYI - we verified our primary design change with the author of Sun's > X509 Trustmanager implementation and the outcome is that what we are > proposing makes sense at the heart of it - "Instead of writing TrustManager > just plugin the Trust store". We are open to discuss additional changes > that you/anybody else would like to see on the functionality however. > > > Thanks > Maulin > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada <maulin.vasav...@gmail.com> > wrote: > >> Hi Harsha >> >> Any response on my question? I feel this KIP is worth accommodating. Your >> help is much appreciated. >> >> Thanks >> Maulin >> >> On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada < >> maulin.vasav...@gmail.com> wrote: >> >>> Hi Harsha >>> >>> I've examined the SPIFFE provider more and have one question - >>> >>> If SPIFFE didn't have a need to do checkSpiffeId() call at the below >>> location, would you really still write the Provider? *OR* >>> Would you just use TrustManagerFactory.init(KeyStore) signature to pass >>> the KeyStore from set of certs returned by spiffeIdManager. >>> getTrustedCerts()? >>> >>> >>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/CertificateUtils.java#L100 >>> >>> >>> /** >>>> * Validates that the SPIFFE ID is present and matches the SPIFFE ID >>>> configured in >>>> * the java.security property ssl.spiffe.accept >>>> * >>>> * If the authorized spiffe ids list is empty any spiffe id is authorized >>>> * >>>> * @param chain an array of X509Certificate that contains the Peer's >>>> SVID to be validated >>>> * @throws CertificateException when either the certificates doesn't >>>> have a SPIFFE ID or the SPIFFE ID is not authorized >>>> */ >>>> static void checkSpiffeId(X509Certificate[] chain) throws >>>> CertificateException { >>> >>> >>> >>> Thanks >>> Maulin >>> >>> >>> On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani <ka...@harsha.io> >>> wrote: >>> >>>> Maulin, >>>> The code parts you are pointing are specific for Spiffe >>>> and if >>>> you are talking about validate method which uses PKIX check like any >>>> other provider does. >>>> If you want to default to SunJSSE everywhere you can do so by delegating >>>> the calls in these methods to SunJSSE provider. >>>> >>>> TrustManagerFactory tmf = TrustManagerFactory >>>> .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use >>>> tmf.chekServerTrusted() >>>> or use >>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/TrustManagerFactory.html#getInstance(java.lang.String)if >>>> you want a specific provider. >>>> >>>> -Harsha >>>> >>>> >>>> On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada < >>>> maulin.vasav...@gmail.com> >>>> wrote: >>>> >>>> > Okay, so I take that you guys agree that I have to write a 'custom' >>>> > algorithm and a provider to make it work , correct? >>>> > >>>> > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an >>>> > implementation per say , ..." , I diagree. You can refer to https:// >>>> > >>>> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/ >>>> > SpiffeTrustManager.java#L91 and >>>> > >>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >>>> > provider/CertificateUtils.java#L100 >>>> > >>>> > "that code" is the customization you have for the custom way to check >>>> > something on top of regular checks. That method is NOT doing custom >>>> > truststore loading. It is validating/verifying something in the >>>> "custom" >>>> > way with spiffeId. >>>> > I bet that without that you won't have a need of the custom algorithm >>>> in >>>> > the first place. >>>> > >>>> > Let me know if you agree to this. >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mpr...@gmail.com> >>>> wrote: >>>> > >>>> > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use >>>> the >>>> > KIP492, you need to create a new provider, register a *new* custom >>>> > algorithm for your keymanager and trustmanager factory >>>> implementations. >>>> > After this, the kafka server configuration can be done as given below >>>> > >>>> > # Register the provider class with custom algorithm, say CUSTOM >>>> security. >>>> > provider.classes=com.company.security.CustomProvider >>>> > <http://security.provider.classes >>>> =com.company.security.customprovider/> >>>> > >>>> > # Register the keymanager and trustmanager algorithms >>>> > # These algorithms indicate that the Keymanager and Trustmanagers >>>> > registered under the algorithm "CUSTOM" needs to be used >>>> > ssl.trustmanager.algorithm=CUSTOM >>>> > ssl.keymanager.algorithm=CUSTOM >>>> > >>>> > And the customprovider looks like this... >>>> > >>>> > public class CustomProvider extends Provider { >>>> > public CustomProvider() { >>>> > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore"); >>>> > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory"); >>>> > super.put("TrustManagerFactory.CUSTOM", >>>> > "customTrustManagerFactory"); >>>> > } >>>> > } >>>> > >>>> > The PR for this is in review and can be found here: >>>> https://github.com/ >>>> > apache/kafka/pull/7090 >>>> > This PR includes the fixed insertProviderAt function call. >>>> > >>>> > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani <ka...@harsha.io> >>>> > wrote: >>>> > >>>> > Answers are in-line >>>> > >>>> > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada < >>>> maulin.vasavada@gmail. >>>> > com >>>> > >>>> > wrote: >>>> > >>>> > Hi Colin >>>> > >>>> > When I refer to "standard" or "custom" algorithms I am following Java >>>> > security Provider Terminology. You can refer to >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ >>>> > StandardNames.html#TrustManagerFactory link I provided earlier in the >>>> > emails. It says PKIX is the default Algorithm for TrustManagerFactory. >>>> > >>>> > 1. For SPIFFE, I am not sure why you are saying 'it does not implement >>>> > custom algorithms' because the following file clearly indicates that >>>> it >>>> > does use custom algorithm- >>>> > >>>> > >>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >>>> > >>>> > provider/SpiffeProvider.java#L17 >>>> > >>>> > Algorithm value: >>>> > >>>> > >>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >>>> > >>>> > provider/SpiffeProviderConstants.java#L6 >>>> > >>>> > @Harsha do you want to chime in since you use that provider? >>>> > >>>> > Here the "Custom" Algorithm is not an implementation per say , rather >>>> > >>>> > used >>>> > >>>> > to invoke the custom trust store factory and key manager factory. You >>>> are >>>> > not moving away from "standard" alogrithms that are available. >>>> > >>>> > >>>> https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >>>> > provider/SpiffeTrustManager.java >>>> > >>>> > As you can see it delegates all the calls of verification of >>>> certificate >>>> > >>>> > to >>>> > >>>> > the default implementation available. >>>> > So in our implementation we still use PKIX to verify the certificate >>>> > chain. So you are not losing anything here and Spiffe is not >>>> reimplementing >>>> > the verification process. >>>> > >>>> > 2. I already mentioned in my 3rd point, in my previous post, why using >>>> > >>>> > ssl.provider does NOT work. I updated KIP-486 in "rejected >>>> > >>>> > alternatives" >>>> > >>>> > also why ssl.provider does not work. >>>> > >>>> > As mentioned before , provider is the right way to go. I am still not >>>> > understanding the gap is. >>>> > If I understand correctly your argument is , provider is going to ask >>>> to >>>> > implement a custom algorithm. >>>> > My answer to that is , no you are not re-implementing the algorithm. >>>> > >>>> > Please >>>> > >>>> > check the above link , TrustManager implementation it delegates the >>>> calls >>>> > back. There is no need to implement your own here. >>>> > >>>> > 3. Security.insertProviderAt() comments were based on assumption if >>>> > >>>> > KIP-492 >>>> > >>>> > changes are done and we use that mechanism to configure providers >>>> > >>>> > instead >>>> > >>>> > of ssl.provider configuration. >>>> > >>>> > KIP-492 has patch available and going through review. >>>> > >>>> > Can you read my all the points, I mentioned in my previous post, very >>>> > >>>> > carefully? I am covering all the aspects in explaining. I am open to >>>> > >>>> > still >>>> > >>>> > discuss more to clarify any doubts. >>>> > >>>> > "3. If we just use existing ssl.provider kafka configuration then our >>>> > provider will be used in SSLContext.getInstance(protocol, provider) >>>> call >>>> > >>>> > in >>>> > >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> >>>> and >>>> > if our provider does not have >>>> > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we >>>> > >>>> > tested >>>> > >>>> > it). Example: In MyProvider sample above you see that I didn't add >>>> > SSLContext.TLSv1 as >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider >>>> you >>>> > don't have this challenge since you are planning to bypass >>>> ssl.provider >>>> > >>>> > as >>>> > >>>> > you mention in the KIP-492." >>>> > >>>> > Yes here you need to pass the protocol that your >>>> KeyManager/TrustManager >>>> > registered with and in no way its deviating from TLS RFC spec. >>>> > >>>> > >>>> https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/ >>>> > javax/net/ssl/SSLContext.java#L134 >>>> > >>>> > My suggestion here is for you to implement a simple Security Provider >>>> as >>>> > you did before and register a Algorithm. You can use the existing >>>> > implementation in SpiffeProvider and plug in changes where you need to >>>> > retrieve the certificates from by making RPC call. >>>> > >>>> > Run an end-to-end test with Kafka broker coming up with your provider >>>> > making calls to RPC call. You do need to pass the "custom algorithm" >>>> that >>>> > you registered in your key manager to invoke the provider. I think >>>> your >>>> > concern is this approach is replacing the existing known ciphersuites >>>> and >>>> > certificate validation provided by java. Which its not. >>>> > >>>> > Now test the TLS connection you can do so via openssl -s_client >>>> options >>>> > >>>> > to >>>> > >>>> > test if everything else is working. >>>> > >>>> > I am happy to share configs that we used if you are interested. >>>> > >>>> > Thanks, >>>> > Harsha >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cmcc...@apache.org> >>>> > >>>> > wrote: >>>> > >>>> > Hi Maulin, >>>> > >>>> > A lot of JSSE providers don't implement custom algorithms. Spire is a >>>> > >>>> > good >>>> > >>>> > example of a JSSE provider that doesn't, and yet is still useful to >>>> > >>>> > many >>>> > >>>> > people. Your JSSE provider can work fine even if it doesn't implement >>>> a >>>> > custom algorithm. >>>> > >>>> > Maybe I'm missing something, but I don't understand the discussion of >>>> > Security.insertProviderAt() that you included. SslEngineBuilder >>>> doesn't >>>> > >>>> > use >>>> > >>>> > that API to get the security provider. Instead, it calls >>>> > "SSLContext.getInstance(protocol, provider)", where provider is the >>>> > >>>> > name >>>> > >>>> > of the provider. >>>> > >>>> > best, >>>> > Colin >>>> > >>>> > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote: >>>> > >>>> > On top of everything above I feel strongly to add the 4th point which >>>> > >>>> > is >>>> > >>>> > based on Java APIs for TrustManagerFactory.init(KeyStore) ( >>>> > >>>> > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/ >>>> > TrustManagerFactory.html#init(java.security.KeyStore >>>> > <http://java.security.keystore/>) >>>> > ) >>>> > >>>> > and KeyManagerFactory.init(KeyStore, char[]) ( >>>> > >>>> > >>>> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory >>>> . >>>> > >>>> > >>>> > html#init(java.security.KeyStore <http://java.security.keystore/ >>>> >,%20char[]) >>>> > >>>> > >>>> > ). >>>> > >>>> > 4. The above APIs are intended to support providing "trust/key >>>> > >>>> > material" >>>> > >>>> > from the user without having to write their own >>>> > >>>> > TrustManager/KeyManagers. >>>> > >>>> > To quote from the TrustManagerFactory.init()'s documentation >>>> > >>>> > "Initializes >>>> > >>>> > this factory with a source of certificate authorities and related >>>> trust >>>> > material." >>>> > To quote from the KeyManagerFactory.init()'s documentation >>>> "Initializes >>>> > this factory with a source of key material." >>>> > >>>> > Based on this it is clear that there is a flexibility provided by Java >>>> > >>>> > to >>>> > >>>> > to enable developers to provide the required trust/key material loaded >>>> > >>>> > from >>>> > >>>> > "anywhere" without requiring them to write custom provider OR >>>> trust/key >>>> > managers. This same flexibility is reflected in Kafka code also where >>>> > >>>> > it >>>> > >>>> > loads the trust/keys from a local file and doesn't require writing a >>>> > Provider necessarily. If we do NOT have a custom algorithm, it makes >>>> > >>>> > less >>>> > >>>> > sense to write a Provider. >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada < >>>> > >>>> > maulin.vasav...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Hi Harsha/Colin >>>> > >>>> > I did the sample with a custom Provider for TrustStoreManager and >>>> tried >>>> > using ssl.provider Kafka config AND the way KIP-492 is suggesting (by >>>> > adding Provider programmatically instead of relying on >>>> > >>>> > ssl.provider+java. >>>> > >>>> > security. The below sample is followed by my detailed findings. I'll >>>> > appreciate if you can go through it carefully and see >>>> > >>>> > if you >>>> > >>>> > see my point. >>>> > >>>> > package providertest; >>>> > >>>> > import java.security.Provider <http://java.security.provider/> >>>> <http:// >>>> > java.security.provider/>; >>>> > >>>> > public class MyProvider extends Provider { >>>> > >>>> > private static final String name = "MyProvider"; private static double >>>> > version = 1.0d; >>>> > private static String info = "Maulin's SSL Provider v"+version; >>>> > >>>> > public MyProvider() { >>>> > super(name, version, info); >>>> > this.put("TrustManagerFactory.PKIX", >>>> > >>>> > "providertest.MyTrustManagerFactory"); >>>> > >>>> > } >>>> > } >>>> > >>>> > *Details:* >>>> > >>>> > KIP-492 documents that it will use Security.addProvider() assuming it >>>> > >>>> > will >>>> > >>>> > add it as position '0' which is not a correct assumption. The >>>> > addProvider()'s documentation says it will add it to the last >>>> available >>>> > position. You may want to correct that to say >>>> > Security.insertProviderAt(provider, 1). >>>> > >>>> > Now coming back to our specific discussion, >>>> > >>>> > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add >>>> > >>>> > that >>>> > >>>> > provider in the provider list via Security.addProvider() the position >>>> > >>>> > where >>>> > >>>> > it gets added doesn't matter (even if you don't end up adding it as >>>> > >>>> > first >>>> > >>>> > entry) since that is the ONLY provider for SPIFFE specific algorithm >>>> > >>>> > you >>>> > >>>> > might have. >>>> > >>>> > We do *not* have custom algorithm for Key/Trust StoreMangers. Which >>>> > >>>> > means >>>> > >>>> > we have to use X509, PKIX etc "Standard Algorithms" (( >>>> > >>>> > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ >>>> > StandardNames.html >>>> > )) >>>> > >>>> > in our provider to override the TrustStoreManager (see my sample code) >>>> > >>>> > and >>>> > >>>> > KeyStoreManger and KeyManager. This creates another challenge >>>> > >>>> > mentioned in >>>> > >>>> > the below point. >>>> > >>>> > 2. In order to make our Provider for loading custom TrustStore work, >>>> we >>>> > have to add the provider as 'first' in the list since there are others >>>> > >>>> > with >>>> > >>>> > the same algorithm. >>>> > >>>> > However, the programatic way of adding provider >>>> > (Security.insertProviderAt()) is *not* deterministic for ordering >>>> since >>>> > different code can call the method for a different provider and >>>> > >>>> > depending >>>> > >>>> > upon the order of the call our provider can be first or pushed down >>>> the >>>> > list. This can happen very well in any client application using Kafka. >>>> > >>>> > This >>>> > >>>> > is specially problematic for a case when you want to guarantee order >>>> > >>>> > for a >>>> > >>>> > Provider having "Standard Algorithms". >>>> > >>>> > If we add our provider in java.security file that definitely >>>> guarantees >>>> > the order(unless somebody calls removeProvider() which is less >>>> > >>>> > likely). But >>>> > >>>> > if we add our provider in java.security file it will defeat the >>>> > >>>> > purpose of >>>> > >>>> > the KIP-492. >>>> > >>>> > In the gist - Apache Kafka must not rely on "un-deterministic" method >>>> > >>>> > to >>>> > >>>> > rely on Provider ordering. >>>> > >>>> > 3. If we just use existing ssl.provider kafka configuration then our >>>> > provider will be used in SSLContext.getInstance(protocol, provider) >>>> > >>>> > call in >>>> > >>>> > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> >>>> and >>>> > if our provider does not have implementation for >>>> > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks >>>> > >>>> > (we >>>> > >>>> > tested it). Example: >>>> > >>>> > In >>>> > >>>> > MyProvider sample above you see that I didn't add SSLContext.TLSv1 as >>>> > "Service+Algorithm" and that didn't work for me. In SPIFFE provider >>>> you >>>> > don't have this challenge since you are planning to bypass >>>> > >>>> > ssl.provider as >>>> > >>>> > you mention in the KIP-492. >>>> > >>>> > *Overall summary,* >>>> > >>>> > 1. Any provider based mechanisms- a) existing ssl.provider and >>>> > >>>> > b)KIP-492, >>>> > >>>> > for loading key/trust store using "Standard Algorithms" do not work >>>> > >>>> > 2. Approach suggested in our KIP-486 works without any issue and it is >>>> > *not* our context specific solve >>>> > >>>> > 3. Based on above we feel KIP-492 and KIP-486 are complimentary >>>> changes >>>> > and not contradicting or redundent. >>>> > >>>> > If you want we can do a joint session somehow to walk through the >>>> > >>>> > sample I >>>> > >>>> > have and various experiments I did. I would encourage you to do >>>> similar >>>> > exercise by writing a Provider for "Standard Algorithm" for >>>> > TrustStoreManager (like our needs) and see what you find since only >>>> > >>>> > writing >>>> > >>>> > samples can bring out the complexity/challenges we face. >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada < >>>> > >>>> > maulin.vasavada@gmail. >>>> > >>>> > com> wrote: >>>> > >>>> > Just to update - still working on it. Get to work only on and off on >>>> > >>>> > it :( >>>> > >>>> > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada < >>>> > >>>> > maulin.vasav...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Hi Harsha >>>> > >>>> > Let me try to write samples and will let you know. >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <harsha...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Hi Maulin, >>>> > With java security providers can be as custom you would >>>> > >>>> > like >>>> > >>>> > it to >>>> > be. If you only want to to implement a custom way of loading the >>>> > >>>> > keystore >>>> > >>>> > and truststore and not implement any protocol/encryption handling you >>>> can >>>> > leave them empty and no need to implement. Have you looked into the >>>> links I >>>> > pasted before? >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > >>>> > >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore. >>>> > >>>> > java >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/ >>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/> >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > >>>> > >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. >>>> > >>>> > java >>>> > >>>> > Can you please tell me which methods are too complex in above to >>>> > >>>> > implement >>>> > >>>> > or unnecessary? You are changing anything in SSL/TLS implementations >>>> > provided by >>>> > >>>> > All of the implementations delegating the checks to the default >>>> > implementation anyway. >>>> > Spire agent is an example, its nothing but a GRPC server listening >>>> > >>>> > on a >>>> > >>>> > unix domain socket . Above code is making a RPC call to the local >>>> > >>>> > daemon >>>> > >>>> > to >>>> > get the certificate and keys. The mechanics are pretty much same as >>>> > >>>> > what >>>> > >>>> > you are asking for. >>>> > >>>> > Thanks, >>>> > Harsha >>>> > >>>> > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada < >>>> > >>>> > maulin.vasav...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Imagine a scenario like - We know we have a custom KMS and as a >>>> > >>>> > Kafka >>>> > >>>> > owner >>>> > >>>> > we want to comply to using that KMS source to load keys/certs. As >>>> > >>>> > a >>>> > >>>> > Kafka >>>> > >>>> > owner we know how to integrate with KMS but doesn't necessarily >>>> > >>>> > have >>>> > >>>> > to >>>> > >>>> > know anything about cipher suites, algorithms, and SSL/TLS >>>> > >>>> > implementation. >>>> > >>>> > Going the Provider way requires to know lot more than we should, >>>> > >>>> > isn't it? >>>> > >>>> > Not that we would have concern/shy-away knowing those details - >>>> > >>>> > but >>>> > >>>> > if we >>>> > >>>> > don't have to - why should we? >>>> > >>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada < >>>> > >>>> > maulin.vasav...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Hi Harsha >>>> > >>>> > We don't have spire (or similar) agents and we do not have >>>> > >>>> > keys/certs >>>> > >>>> > locally on any brokers. To elaborate more on my previous email, >>>> > >>>> > I agree that Java security Providers are used in much broader >>>> > >>>> > sense >>>> > >>>> > - to >>>> > >>>> > have a particular implementation of an algorithm, use specific >>>> > >>>> > cipher >>>> > >>>> > suites for SSL , OR in our current team's case have a >>>> > >>>> > particular >>>> > >>>> > way to >>>> > >>>> > leverage pre-generated SSL sessions. However, the scope of our >>>> > >>>> > KIP >>>> > >>>> > (486) >>>> > >>>> > is >>>> > >>>> > much restricted than that. We merely intend to provide a custom >>>> > keystore/truststore for our SSL connections and not really worry >>>> > >>>> > about >>>> > >>>> > underlying specific SSL/TLS implementation. This simplifies it >>>> > >>>> > a >>>> > >>>> > lot for >>>> > >>>> > us to keep the concerns separate and clear. >>>> > >>>> > I feel our approach is more complimentary such that it allows >>>> > >>>> > for >>>> > >>>> > using >>>> > >>>> > keystores of choice while retaining the flexibility to use any >>>> > underlying/available Provider for actually making the SSL call. >>>> > >>>> > We agree with KIP-492's approach based on Providers (and Java's >>>> > recommendation), but also strongly believe that our approach can >>>> > >>>> > compliment >>>> > >>>> > it very effectively for reasons explained above. >>>> > >>>> > Thanks >>>> > Maulin >>>> > >>>> > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani < >>>> > >>>> > ka...@harsha.io >>>> > >>>> > wrote: >>>> > >>>> > Hi Maulin, >>>> > >>>> > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada < >>>> > >>>> > maulin.vasavada@gmail. >>>> > >>>> > com> >>>> > wrote: >>>> > >>>> > Hi Harsha >>>> > >>>> > The reason we rejected the SslProvider route is that - we >>>> > >>>> > only >>>> > >>>> > needed >>>> > >>>> > a >>>> > >>>> > custom way to load keys/certs. Not touch any policy that >>>> > >>>> > existing >>>> > >>>> > Providers >>>> > >>>> > govern like SunJSSE Provider. >>>> > >>>> > We have exactly the same requirements to load certs and keys >>>> > >>>> > through >>>> > >>>> > spire >>>> > >>>> > agent. We used security.provider to do that exactly. I am not >>>> > >>>> > sure >>>> > >>>> > why >>>> > >>>> > you >>>> > >>>> > would be modifying any policies provided by default SunJSSE >>>> > >>>> > provider. >>>> > >>>> > Can >>>> > >>>> > you give me an example of having custom provider that will >>>> > >>>> > override an >>>> > >>>> > existing policy in SunJSSE provider. >>>> > >>>> > As pointed out earlier, this kip >>>> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/ >>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config >>>> > >>>> > allows >>>> > you to load security.provider through config. >>>> > Take a look at the examples I gave before >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > >>>> > >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. >>>> > >>>> > java >>>> > >>>> > It registers KeyManagerFactory and TrustManagerFactory and >>>> > >>>> > Keystore >>>> > >>>> > algorithm. >>>> > Implement your custom way of loading Keystore in here >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > >>>> > >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore. >>>> > >>>> > java >>>> > >>>> > and Trust manager like here >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > spiffe-security-provider/src/main/java/spiffe/api/provider/ >>>> > SpiffeTrustManager.java <http://spiffetrustmanager.java/> >>>> > >>>> > In your Kafka client you can set the security.provider to your >>>> > >>>> > custom >>>> > >>>> > implementation and with this fix >>>> > https://issues.apache.org/jira/browse/KAFKA-8191 you can set >>>> > keyManagerAlgorigthm and trustManagerAlgorithm configs. >>>> > >>>> > All of this is in your clients and broker side and do not need >>>> > >>>> > to >>>> > >>>> > touch >>>> > >>>> > any >>>> > policy changes at JVM level. You'll register the providers in >>>> > >>>> > the >>>> > >>>> > priority >>>> > >>>> > order and can still have SunJSSE provider and have your custom >>>> > >>>> > provider >>>> > >>>> > to >>>> > >>>> > implement the key and trust managers. >>>> > >>>> > The ask here is different than KIP-492. We don't have any need >>>> > >>>> > to >>>> > >>>> > modify/specify the algorithm parameter. Does that make sense? >>>> > >>>> > The ask in KIP is introducing new interfaces where the KIP's >>>> > goal/motivation can be achieved through the security.provider >>>> > >>>> > and >>>> > >>>> > we >>>> > >>>> > worked >>>> > on similar goal without touching any Keystore or Truststore >>>> > >>>> > interfaces. >>>> > >>>> > My advise is against changing or introducing new interfaces >>>> > >>>> > when >>>> > >>>> > it can >>>> > >>>> > work through security.provider. >>>> > >>>> > Thanks, >>>> > Harsha >>>> > >>>> > Thanks >>>> > >>>> > Maulin >>>> > >>>> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani < >>>> > >>>> > ka...@harsha.io> >>>> > >>>> > wrote: >>>> > >>>> > In your KIP you added security. provider as rejected >>>> > >>>> > alternative >>>> > >>>> > and >>>> > >>>> > specified "its not the correct way". Do you mind explaining >>>> > >>>> > why >>>> > >>>> > its >>>> > >>>> > not? I >>>> > >>>> > didn't find any evidence in Java docs to say so. Contrary to >>>> > >>>> > your >>>> > >>>> > statement >>>> > >>>> > it does say in the java docs >>>> > " However, please note that a provider can be used to >>>> > >>>> > implement >>>> > >>>> > any >>>> > >>>> > security service in Java that uses a pluggable architecture >>>> > >>>> > with >>>> > >>>> > a >>>> > >>>> > choice >>>> > >>>> > of implementations that fit underneath." >>>> > >>>> > Java Security Providers have been used by other projects to >>>> > >>>> > provide >>>> > >>>> > such >>>> > >>>> > integration . I am not sure if you looked into Spiffe >>>> > >>>> > project to >>>> > >>>> > efficiently distribute certificates but here is an example of >>>> > >>>> > Java >>>> > >>>> > provider >>>> > >>>> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >>>> > >>>> > >>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. >>>> > >>>> > java which >>>> > obtains certificates from local daemons. >>>> > These integrations are being used in Tomcat, Jetty etc.. We >>>> > >>>> > are >>>> > >>>> > also >>>> > >>>> > using >>>> > >>>> > Security provider to do the same in our Kafka clusters. So >>>> > >>>> > unless I >>>> > >>>> > see >>>> > >>>> > more evidence why security.provider doesn't work for you >>>> > >>>> > adding >>>> > >>>> > new >>>> > >>>> > interfaces while there exists more cleaner way of achieving >>>> > >>>> > the >>>> > >>>> > goals >>>> > >>>> > of >>>> > >>>> > this KIP is unnecessary and breaks the well known security >>>> > >>>> > interfaces >>>> > >>>> > provided by Java itself. >>>> > >>>> > Thanks, >>>> > Harsha >>>> > >>>> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani < >>>> > >>>> > ka...@harsha.io >>>> > >>>> > wrote: >>>> > >>>> > Hi Maulin, >>>> > Not sure if you looked at my previous replies. This >>>> > >>>> > changes >>>> > >>>> > are not required as there is already security Provider to do >>>> > >>>> > what you >>>> > >>>> > are >>>> > >>>> > proposing. This KIP >>>> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/ >>>> > >>>> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config >>>> > >>>> > also >>>> > >>>> > addresses easy registration of such providers. >>>> > >>>> > Thanks, >>>> > Harsha >>>> > >>>> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada >>>> > >>>> > <maulin.vasavada@gmail. >>>> > >>>> > com> wrote: >>>> > >>>> > Bump! Can somebody please review this? >>>> > >>>> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada < >>>> > >>>> > maulin.vasav...@gmail.com> >>>> > >>>> > wrote: >>>> > >>>> > Bump! Can somebody please review this? >>>> > >>>> > -- >>>> > Thanks, >>>> > M.Sai Sandeep >>>> > >>>> > >>>> >>>
