[jira] [Commented] (KAFKA-6886) Externalize Secrets for Kafka Connect Configurations

2018-05-30 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495724#comment-16495724
 ] 

ASF GitHub Bot commented on KAFKA-6886:
---

ewencp closed pull request #5068: KAFKA-6886 Externalize secrets from Connect 
configs
URL: https://github.com/apache/kafka/pull/5068
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/checkstyle/suppressions.xml b/checkstyle/suppressions.xml
index ba48c38cb28..5bf69b6b65f 100644
--- a/checkstyle/suppressions.xml
+++ b/checkstyle/suppressions.xml
@@ -83,7 +83,7 @@
   
files="(KafkaConfigBackingStore|RequestResponseTest|WorkerSinkTaskTest).java"/>
 
 
+  files="(WorkerSinkTask|WorkerSourceTask).java"/>
 
 http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config;
+
+/**
+ * A callback passed to {@link ConfigProvider} for subscribing to changes.
+ */
+public interface ConfigChangeCallback {
+
+/**
+ * Performs an action when configuration data changes.
+ *
+ * @param path the path at which the data resides
+ * @param data the configuration data
+ */
+void onChange(String path, ConfigData data);
+}
diff --git 
a/clients/src/main/java/org/apache/kafka/common/config/ConfigData.java 
b/clients/src/main/java/org/apache/kafka/common/config/ConfigData.java
new file mode 100644
index 000..2bd0ff6b06a
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/config/ConfigData.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config;
+
+import java.util.Map;
+
+/**
+ * Configuration data from a {@link ConfigProvider}.
+ */
+public class ConfigData {
+
+private final Map data;
+private final Long ttl;
+
+/**
+ * Creates a new ConfigData with the given data and TTL (in milliseconds).
+ *
+ * @param data a Map of key-value pairs
+ * @param ttl the time-to-live of the data in milliseconds, or null if 
there is no TTL
+ */
+public ConfigData(Map data, Long ttl) {
+this.data = data;
+this.ttl = ttl;
+}
+
+/**
+ * Creates a new ConfigData with the given data.
+ *
+ * @param data a Map of key-value pairs
+ */
+public ConfigData(Map data) {
+this(data, null);
+}
+
+/**
+ * Returns the data.
+ *
+ * @return data a Map of key-value pairs
+ */
+public Map data() {
+return data;
+}
+
+/**
+ * Returns the TTL (in milliseconds).
+ *
+ * @return ttl the time-to-live (in milliseconds) of the data, or null if 
there is no TTL
+ */
+public Long ttl() {
+return ttl;
+}
+}
diff --git 
a/clients/src/main/java/org/apache/kafka/common/config/ConfigProvider.java 
b/clients/src/main/java/org/apache/kafka/common/config/ConfigProvider.java
new file mode 100644
index 000..7133baaebd0
--- /dev/null
+++ b/clients/src/main/java/org/apache/kafka/common/config/ConfigProvider.java
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License 

[jira] [Commented] (KAFKA-6886) Externalize Secrets for Kafka Connect Configurations

2018-05-22 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486495#comment-16486495
 ] 

ASF GitHub Bot commented on KAFKA-6886:
---

rayokota opened a new pull request #5068: KAFKA-6886 Externalize secrets from 
Connect configs
URL: https://github.com/apache/kafka/pull/5068
 
 
   This commit allows secrets in Connect configs to be externalized and 
replaced with variable references of the form `${provider:[path:]key}`.  
   
   There are 2 main additions to `org.apache.kafka.common.config`: a 
`ConfigProvider` and a `ConfigTransformer`.  The `ConfigProvider` is an 
interface that allows key-value pairs to be provided by an external source for 
a given "path".  An a TTL can be associated with the key-value pairs returned 
from a "path".  The `ConfigTransformer` will use instances of `ConfigProvider` 
to replace variable references in a set of configuration values.
   
   In the Connect framework, `ConfigProvider` classes can be specified in the 
worker config, and then variable references can be used in the connector 
config.  In addition, the herder can be configured to restart connectors (or 
not) based on the TTL returned from a `ConfigProvider`.  The main class that 
performs restarts and transformations is `WorkerConfigTransformer`.  
   
   Finally, a `configs()` method has been added to both `SourceTaskContext` and 
`SinkTaskContext`.  This allows connectors to get configs with variables 
replaced by the latest values from instances of `ConfigProvider`.
   
   Most of the other changes in the Connect framework are threading various 
objects through classes to enable the above functionality.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Externalize Secrets for Kafka Connect Configurations
> 
>
> Key: KAFKA-6886
> URL: https://issues.apache.org/jira/browse/KAFKA-6886
> Project: Kafka
>  Issue Type: New Feature
>  Components: KafkaConnect
>Reporter: Robert Yokota
>Assignee: Robert Yokota
>Priority: Major
> Fix For: 2.0.0
>
>
> Kafka Connect's connector configurations have plaintext passwords, and 
> Connect stores these in cleartext either on the filesystem (for standalone 
> mode) or in internal topics (for distributed mode). 
> Connect should not store or transmit cleartext passwords in connector 
> configurations. Secrets in stored connector configurations should be allowed 
> to be replaced with references to values stored in external secret management 
> systems. Connect should provide an extension point for adding customized 
> integrations, as well as provide a file-based extension as an example. 
> Second, a Connect runtime should be allowed to be configured to use one or 
> more of these extensions, and allow connector configurations to use 
> placeholders that will be resolved by the runtime before passing the complete 
> connector configurations to connectors. This will allow existing connectors 
> to not see any difference in the configurations that Connect provides to them 
> at startup. And third, Connect's API should be changed to allow a connector 
> to obtain the latest connector configuration at any time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)