Randall Hauch created KAFKA-6142:
------------------------------------
Summary: Connect worker configurations and connector
configurations should accept environment variables
Key: KAFKA-6142
URL: https://issues.apache.org/jira/browse/KAFKA-6142
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Affects Versions: 1.0.0
Reporter: Randall Hauch
Currently, when a worker or connector configuration is parsed, the values are
used as-is without any kind of pre-processing before the value is used. It
should be possible to define configuration properties such that string literal
values or default values can use *_configuration variables_* that reference
environment variables and/or system properties, and that these configuration
variables are resolved/replaced before the configuration value is used.
I propose doing this enhancement in Kafka client's {{ConfigDef}} by adding a
{{ConfigDef.Transformer}} interface:
{code:java}
/**
* Transform the configuration value.
*/
public interface Transformer {
/**
* Transform the configuration value.
* @param name The name of the configuration
* @param value The value of the configuration
* @return the preprocessed value
* @throws ConfigException if the value is invalid.
*/
Object apply(String name, Object value);
}
{code}
and then allowing {{Transformer}} implementations to be passed to
{{ConfigDef.define(...)}} such all existing signatures are maintained for
backward compatibility. By default, the definition would use an identity
transform that simply returns the value. The transformers would be called in
{{ConfigDef.parseValue(...)}} before the {{parseType(...)}} method is called,
and would also be called on the default value if one is provided.
Then, a {{ConfigDef.ReplaceSystemVariables}} implementation would be provided
to look in {{String}} values for zero or more variables defined with this EBNF
grammar:
{noformat}
'$' '{' varName { ',' varName } [ ':' defaultValue] '}'
{noformat}
where:
* {{varName}} is the name of a Java system property or {{env.}} followed by the
name of an environment variable, and
* {{defaultValue}} specifies the replacement value used when no environment
variable or system property is found, and defaults to an empty string.
The value of the first system property or environment variable resolved is then
used to replace the variable expression. This implementation would have trace
or debug level logging to describe what it is doing.
For example, the variable {{$\{env.KAFKA_HOME\}}} would replace the variable
expression with the value of the {{KAFKA_HOME}} environment variable or an
blank string if that variable doesn't exist.
Likewise, the variable {{$\{foo.prop1,foo.prop2,env.MY_ENV_VAR:value\}}} would
be replaced with the value of the {{foo.prop1}} system property if it exists,
or with the value of the {{foo.prop2}} system property if it exists, or with
the value of the {{MY_ENV_VAR}} environment variable if it exists, or {{value}}
if none of the system properties exist.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
