[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779049#comment-16779049
]
Valeria Vasylieva commented on KAFKA-6755:
--
Thank you for your help Randall!
I have created wiki account:
[https://cwiki.apache.org/confluence/display/~nimfadora]
Once the KIP will be created, I will be happy if you review it.
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780554#comment-16780554
]
Valeria Vasylieva edited comment on KAFKA-6755 at 2/28/19 2:15 PM:
---
[~rhauch] I have created [KIP
draft|https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT]
It would be really helpful if you could look at it and share your opinion.
Thank you!
was (Author: nimfadora):
[~rhauch] I have created [KIP
draft|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT].]
It would be really helpful if you could look at it and share your opinion.
Thank you!
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780554#comment-16780554
]
Valeria Vasylieva commented on KAFKA-6755:
--
[~rhauch] I have created [KIP
draft|[https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT].]
It would be really helpful if you could look at it and share your opinion.
Thank you!
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7157) Connect TimestampConverter SMT doesn't handle null values
[
https://issues.apache.org/jira/browse/KAFKA-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781704#comment-16781704
]
Valeria Vasylieva commented on KAFKA-7157:
--
Issue can be still reproduced, [~rhauch] I'll fix it, if you don't mind.
> Connect TimestampConverter SMT doesn't handle null values
> -
>
> Key: KAFKA-7157
> URL: https://issues.apache.org/jira/browse/KAFKA-7157
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Randall Hauch
>Priority: Major
>
> TimestampConverter SMT is not able to handle null values (in any versions),
> so it's always trying to apply the transformation to the value. Instead, it
> needs to check for null and use the default value for the new schema's field.
> {noformat}
> [2018-07-03 02:31:52,490] ERROR Task MySourceConnector-2 threw an uncaught
> and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.transforms.TimestampConverter$2.toRaw(TimestampConverter.java:137)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.convertTimestamp(TimestampConverter.java:440)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.applyValueWithSchema(TimestampConverter.java:368)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.applyWithSchema(TimestampConverter.java:358)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.apply(TimestampConverter.java:275)
>
> at
> org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:38)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:435)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:264)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:182)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:150)
>
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> [2018-07-03 02:31:52,491] ERROR Task is being killed and will not recover
> until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7157) Connect TimestampConverter SMT doesn't handle null values
[
https://issues.apache.org/jira/browse/KAFKA-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valeria Vasylieva reassigned KAFKA-7157:
Assignee: Valeria Vasylieva
> Connect TimestampConverter SMT doesn't handle null values
> -
>
> Key: KAFKA-7157
> URL: https://issues.apache.org/jira/browse/KAFKA-7157
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
>
> TimestampConverter SMT is not able to handle null values (in any versions),
> so it's always trying to apply the transformation to the value. Instead, it
> needs to check for null and use the default value for the new schema's field.
> {noformat}
> [2018-07-03 02:31:52,490] ERROR Task MySourceConnector-2 threw an uncaught
> and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.transforms.TimestampConverter$2.toRaw(TimestampConverter.java:137)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.convertTimestamp(TimestampConverter.java:440)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.applyValueWithSchema(TimestampConverter.java:368)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.applyWithSchema(TimestampConverter.java:358)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.apply(TimestampConverter.java:275)
>
> at
> org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:38)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:435)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:264)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:182)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:150)
>
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> [2018-07-03 02:31:52,491] ERROR Task is being killed and will not recover
> until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789383#comment-16789383
]
Valeria Vasylieva commented on KAFKA-6755:
--
[~rhauch] thanks for the review! I have introduced suggested changes and
started the discussion on the KIP. Hope it will not take long time.
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7157) Connect TimestampConverter SMT doesn't handle null values
[
https://issues.apache.org/jira/browse/KAFKA-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16792870#comment-16792870
]
Valeria Vasylieva commented on KAFKA-7157:
--
[~rhauch] I have submitted PR fix for this issue, I will be very glad if you
look at it. Thank you!
> Connect TimestampConverter SMT doesn't handle null values
> -
>
> Key: KAFKA-7157
> URL: https://issues.apache.org/jira/browse/KAFKA-7157
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
>
> TimestampConverter SMT is not able to handle null values (in any versions),
> so it's always trying to apply the transformation to the value. Instead, it
> needs to check for null and use the default value for the new schema's field.
> {noformat}
> [2018-07-03 02:31:52,490] ERROR Task MySourceConnector-2 threw an uncaught
> and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.transforms.TimestampConverter$2.toRaw(TimestampConverter.java:137)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.convertTimestamp(TimestampConverter.java:440)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.applyValueWithSchema(TimestampConverter.java:368)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.applyWithSchema(TimestampConverter.java:358)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.apply(TimestampConverter.java:275)
>
> at
> org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:38)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:435)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:264)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:182)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:150)
>
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> [2018-07-03 02:31:52,491] ERROR Task is being killed and will not recover
> until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794890#comment-16794890
]
Valeria Vasylieva commented on KAFKA-6755:
--
[~rhauch] KIP discussion has only one commenter, could you advise what to do
to bring attention to it? Thank you!
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-6675) Connect workers should log plugin path and available plugins more clearly
[
https://issues.apache.org/jira/browse/KAFKA-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valeria Vasylieva reassigned KAFKA-6675:
Assignee: Valeria Vasylieva
> Connect workers should log plugin path and available plugins more clearly
> -
>
> Key: KAFKA-6675
> URL: https://issues.apache.org/jira/browse/KAFKA-6675
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.1
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Minor
>
> Users struggle with setting the plugin path and properly installing plugins.
> If users get any of this wrong, they get strange errors only after they run
> the worker and attempt to deploy connectors or use transformations.
> The Connect worker should more obviously output the plugin path directories
> and the available plugins. For example, if the {{plugin.path}} were:
> {code}
> plugin.path=/usr/local/share/java,/usr/local/plugins
> {code}
> then the worker might output something like the following information in the
> log:
> {noformat}
> Looking for plugins on classpath and inside plugin.path directories:
> /usr/local/share/java
> /usr/local/plugins
>
> Source Connector(s):
> FileStreamSource (org.apache.kafka.connect.file.FileStreamSourceConnector)
> @ classpath
> FileStreamSink(org.apache.kafka.connect.file.FileStreamSinkConnector)
> @ classpath
> JdbcSource(io.confluent.connect.jdbc.JdbcSourceConnector)
> @ /usr/local/share/java/kafka-connect-jdbc
> MySql (io.debezium.connector.mysql.MySqlConnector)
> @ /usr/local/plugins/debezium-connector-mysql
> Converter(s):
> JsonConverter (org.apache.kafka.connect.json.JsonConverter)
> @ classpath
> ByteArrayConverter
> (org.apache.kafka.connect.converters.ByteArrayConverter)@ classpath
> SimpleHeaderConverter
> (org.apache.kafka.connect.converters.SimpleHeaderConverter) @ classpath
> AvroConverter (io.confluent.connect.avro.AvroConverter)
> @ /usr/local/share/java/kafka-serde-tools
> Transformation(s):
> InsertField (org.apache.kafka.connect.transforms.InsertField)
> @ classpath
> ReplaceField (org.apache.kafka.connect.transforms.ReplaceField)
> @ classpath
> MaskField (org.apache.kafka.connect.transforms.MaskField)
> @ classpath
> ValueToKey(org.apache.kafka.connect.transforms.ValueToKey)
> @ classpath
> HoistField(org.apache.kafka.connect.transforms.HoistField)
> @ classpath
> ExtractField (org.apache.kafka.connect.transforms.ExtractField)
> @ classpath
> SetSchemaMetadata (org.apache.kafka.connect.transforms.SetSchemaMetadata)
> @ classpath
> RegexRouter (org.apache.kafka.connect.transforms.RegexRouter)
> @ classpath
> TimestampRouter (org.apache.kafka.connect.transforms.TimestampRouter)
> @ classpath
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6675) Connect workers should log plugin path and available plugins more clearly
[
https://issues.apache.org/jira/browse/KAFKA-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795837#comment-16795837
]
Valeria Vasylieva commented on KAFKA-6675:
--
I would like to work on it
> Connect workers should log plugin path and available plugins more clearly
> -
>
> Key: KAFKA-6675
> URL: https://issues.apache.org/jira/browse/KAFKA-6675
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.1
>Reporter: Randall Hauch
>Priority: Minor
>
> Users struggle with setting the plugin path and properly installing plugins.
> If users get any of this wrong, they get strange errors only after they run
> the worker and attempt to deploy connectors or use transformations.
> The Connect worker should more obviously output the plugin path directories
> and the available plugins. For example, if the {{plugin.path}} were:
> {code}
> plugin.path=/usr/local/share/java,/usr/local/plugins
> {code}
> then the worker might output something like the following information in the
> log:
> {noformat}
> Looking for plugins on classpath and inside plugin.path directories:
> /usr/local/share/java
> /usr/local/plugins
>
> Source Connector(s):
> FileStreamSource (org.apache.kafka.connect.file.FileStreamSourceConnector)
> @ classpath
> FileStreamSink(org.apache.kafka.connect.file.FileStreamSinkConnector)
> @ classpath
> JdbcSource(io.confluent.connect.jdbc.JdbcSourceConnector)
> @ /usr/local/share/java/kafka-connect-jdbc
> MySql (io.debezium.connector.mysql.MySqlConnector)
> @ /usr/local/plugins/debezium-connector-mysql
> Converter(s):
> JsonConverter (org.apache.kafka.connect.json.JsonConverter)
> @ classpath
> ByteArrayConverter
> (org.apache.kafka.connect.converters.ByteArrayConverter)@ classpath
> SimpleHeaderConverter
> (org.apache.kafka.connect.converters.SimpleHeaderConverter) @ classpath
> AvroConverter (io.confluent.connect.avro.AvroConverter)
> @ /usr/local/share/java/kafka-serde-tools
> Transformation(s):
> InsertField (org.apache.kafka.connect.transforms.InsertField)
> @ classpath
> ReplaceField (org.apache.kafka.connect.transforms.ReplaceField)
> @ classpath
> MaskField (org.apache.kafka.connect.transforms.MaskField)
> @ classpath
> ValueToKey(org.apache.kafka.connect.transforms.ValueToKey)
> @ classpath
> HoistField(org.apache.kafka.connect.transforms.HoistField)
> @ classpath
> ExtractField (org.apache.kafka.connect.transforms.ExtractField)
> @ classpath
> SetSchemaMetadata (org.apache.kafka.connect.transforms.SetSchemaMetadata)
> @ classpath
> RegexRouter (org.apache.kafka.connect.transforms.RegexRouter)
> @ classpath
> TimestampRouter (org.apache.kafka.connect.transforms.TimestampRouter)
> @ classpath
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6675) Connect workers should log plugin path and available plugins more clearly
[
https://issues.apache.org/jira/browse/KAFKA-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795993#comment-16795993
]
Valeria Vasylieva edited comment on KAFKA-6675 at 3/19/19 11:30 AM:
[~rhauch] I have investigated the actual log:
if we have such configuration:
{code:java}
plugin.path=/etc/kafka/kafka_2.11-2.1.0/connect-plugins,
/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium
{code}
then, we have such output for plugin.path plugins:
{code:java}
[2019-03-19 13:32:47,277] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-cassandra-1.0.2/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,278] INFO Added plugin
'io.confluent.connect.cassandra.CassandraSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,279] INFO Loading plugin from:
/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:47,694] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,694] INFO Added plugin
'io.confluent.connect.avro.AvroConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,695] INFO Loading plugin from:
/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium/debezium-connector-mysql
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:48,083] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium/debezium-connector-mysql/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:48,090] INFO Added plugin
'io.debezium.connector.mysql.MySqlConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:50,128] INFO Registered loader:
sun.misc.Launcher$AppClassLoader@764c12b6
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:50,131] INFO Added plugin
'org.apache.kafka.connect.tools.MockSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
...
[2019-03-19 13:32:50,161] INFO Added aliases 'CassandraSinkConnector' and
'CassandraSink' to plugin
'io.confluent.connect.cassandra.CassandraSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
[2019-03-19 13:32:50,161] INFO Added aliases 'MySqlConnector' and 'MySql' to
plugin 'io.debezium.connector.mysql.MySqlConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390){code}
and such output for classpath plugins:
{code:java}
[2019-03-19 13:32:50,131] INFO Added plugin
'org.apache.kafka.connect.tools.MockSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:50,132] INFO Added plugin
'org.apache.kafka.connect.tools.MockConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
...
[2019-03-19 13:32:50,163] INFO Added aliases 'MockConnector' and 'Mock' to
plugin 'org.apache.kafka.connect.tools.MockConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
[2019-03-19 13:32:50,164] INFO Added aliases 'MockSinkConnector' and 'MockSink'
to plugin 'org.apache.kafka.connect.tools.MockSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
{code}
What is the goal of this task, to make the output more clear and pretty? Almost
all the information (including path), that you stated is already available in
the log.
was (Author: nimfadora):
[~rhauch] I have investigated the actual log:
if we have such configuration:
{code:java}
plugin.path=/etc/kafka/kafka_2.11-2.1.0/connect-plugins,
/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium
{code}
then, we have such output for plugin.path plugins:
{code:java}
[2019-03-19 13:32:47,277] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-cassandra-1.0.2/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,278] INFO Added plugin
'io.confluent.connect.cassandra.CassandraSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,279] INFO Loading plugin from:
/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:47,694] INFO Registered loader
[jira] [Commented] (KAFKA-6675) Connect workers should log plugin path and available plugins more clearly
[
https://issues.apache.org/jira/browse/KAFKA-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795993#comment-16795993
]
Valeria Vasylieva commented on KAFKA-6675:
--
[~rhauch] I have investigated the actual log:
if we have such configuration:
{code:java}
plugin.path=/etc/kafka/kafka_2.11-2.1.0/connect-plugins,
/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium
{code}
then, we have such output for plugin.path plugins:
{code:java}
[2019-03-19 13:32:47,277] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-cassandra-1.0.2/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,278] INFO Added plugin
'io.confluent.connect.cassandra.CassandraSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,279] INFO Loading plugin from:
/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:47,694] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,694] INFO Added plugin
'io.confluent.connect.avro.AvroConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,695] INFO Loading plugin from:
/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium/debezium-connector-mysql
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:48,083] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium/debezium-connector-mysql/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:48,090] INFO Added plugin
'io.debezium.connector.mysql.MySqlConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:50,128] INFO Registered loader:
sun.misc.Launcher$AppClassLoader@764c12b6
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:50,131] INFO Added plugin
'org.apache.kafka.connect.tools.MockSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
...
[2019-03-19 13:32:50,161] INFO Added aliases 'CassandraSinkConnector' and
'CassandraSink' to plugin
'io.confluent.connect.cassandra.CassandraSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
[2019-03-19 13:32:50,161] INFO Added aliases 'MySqlConnector' and 'MySql' to
plugin 'io.debezium.connector.mysql.MySqlConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390){code}
and such output for classpath plugins:
{code:java}
[2019-03-19 13:32:50,131] INFO Added plugin
'org.apache.kafka.connect.tools.MockSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:50,132] INFO Added plugin
'org.apache.kafka.connect.tools.MockConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
...
[2019-03-19 13:32:50,163] INFO Added aliases 'MockConnector' and 'Mock' to
plugin 'org.apache.kafka.connect.tools.MockConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
[2019-03-19 13:32:50,164] INFO Added aliases 'MockSinkConnector' and 'MockSink'
to plugin 'org.apache.kafka.connect.tools.MockSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
{code}
What is the goal of this task, to make the output more clear and pretty? Almost
all the information (including path), that you stated is already available in
the log.
> Connect workers should log plugin path and available plugins more clearly
> -
>
> Key: KAFKA-6675
> URL: https://issues.apache.org/jira/browse/KAFKA-6675
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.1
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Minor
>
> Users struggle with setting the plugin path and properly installing plugins.
> If users get any of this wrong, they get strange errors only after they run
> the worker and attempt to deploy connectors or use transformations.
> The Connect worker should more obviously output the plugin path directories
> and the available plugins. For example, if the {{plugin.path}} were:
> {code}
> plugin.path=/usr/local/share/java,/usr/local/plugins
> {code}
> then the worker might output something like the following information in the
>
[jira] [Comment Edited] (KAFKA-6675) Connect workers should log plugin path and available plugins more clearly
[
https://issues.apache.org/jira/browse/KAFKA-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795993#comment-16795993
]
Valeria Vasylieva edited comment on KAFKA-6675 at 3/19/19 11:39 AM:
[~rhauch] I have investigated the actual log:
if we have such configuration:
{code:java}
plugin.path=/etc/kafka/kafka_2.11-2.1.0/connect-plugins,
/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium
{code}
then, we have such output for plugin.path plugins:
{code:java}
[2019-03-19 13:32:47,277] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-cassandra-1.0.2/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,278] INFO Added plugin
'io.confluent.connect.cassandra.CassandraSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,279] INFO Loading plugin from:
/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:47,694] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,694] INFO Added plugin
'io.confluent.connect.avro.AvroConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,695] INFO Loading plugin from:
/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium/debezium-connector-mysql
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:48,083] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium/debezium-connector-mysql/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:48,090] INFO Added plugin
'io.debezium.connector.mysql.MySqlConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:50,128] INFO Registered loader:
sun.misc.Launcher$AppClassLoader@764c12b6
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:50,131] INFO Added plugin
'org.apache.kafka.connect.tools.MockSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
...
[2019-03-19 13:32:50,161] INFO Added aliases 'CassandraSinkConnector' and
'CassandraSink' to plugin
'io.confluent.connect.cassandra.CassandraSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
[2019-03-19 13:32:50,161] INFO Added aliases 'MySqlConnector' and 'MySql' to
plugin 'io.debezium.connector.mysql.MySqlConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390){code}
and such output for classpath plugins:
{code:java}
[2019-03-19 13:32:50,131] INFO Added plugin
'org.apache.kafka.connect.tools.MockSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:50,132] INFO Added plugin
'org.apache.kafka.connect.tools.MockConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
...
[2019-03-19 13:32:50,163] INFO Added aliases 'MockConnector' and 'Mock' to
plugin 'org.apache.kafka.connect.tools.MockConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
[2019-03-19 13:32:50,164] INFO Added aliases 'MockSinkConnector' and 'MockSink'
to plugin 'org.apache.kafka.connect.tools.MockSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:390)
{code}
Could you please clarify the goal of this task? Almost all the information
(including path), that you stated is already available in the log. Or the goal
is to make the output more clear and pretty?
was (Author: nimfadora):
[~rhauch] I have investigated the actual log:
if we have such configuration:
{code:java}
plugin.path=/etc/kafka/kafka_2.11-2.1.0/connect-plugins,
/etc/kafka/kafka_2.11-2.1.0/connect-plugins-debezium
{code}
then, we have such output for plugin.path plugins:
{code:java}
[2019-03-19 13:32:47,277] INFO Registered loader:
PluginClassLoader{pluginLocation=file:/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-cassandra-1.0.2/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:243)
[2019-03-19 13:32:47,278] INFO Added plugin
'io.confluent.connect.cassandra.CassandraSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:172)
[2019-03-19 13:32:47,279] INFO Loading plugin from:
/etc/kafka/kafka_2.11-2.1.0/connect-plugins/confluentinc-kafka-connect-avro-converter-5.1.2
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:220)
[2019-03-19 13:32:47,69
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16813197#comment-16813197
]
Valeria Vasylieva commented on KAFKA-6755:
--
[~rhauch] discussion did not gain lot of interest from the community, should I
proceed to vote, so that we finally decide if this change should be merged or
discarded?
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6675) Connect workers should log plugin path and available plugins more clearly
[
https://issues.apache.org/jira/browse/KAFKA-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16813204#comment-16813204
]
Valeria Vasylieva commented on KAFKA-6675:
--
[~rhauch] thank you for clarification, I got it and would proceed with the task.
> Connect workers should log plugin path and available plugins more clearly
> -
>
> Key: KAFKA-6675
> URL: https://issues.apache.org/jira/browse/KAFKA-6675
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.1
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Minor
>
> Users struggle with setting the plugin path and properly installing plugins.
> If users get any of this wrong, they get strange errors only after they run
> the worker and attempt to deploy connectors or use transformations.
> The Connect worker should more obviously output the plugin path directories
> and the available plugins. For example, if the {{plugin.path}} were:
> {code}
> plugin.path=/usr/local/share/java,/usr/local/plugins
> {code}
> then the worker might output something like the following information in the
> log:
> {noformat}
> Looking for plugins on classpath and inside plugin.path directories:
> /usr/local/share/java
> /usr/local/plugins
>
> Source Connector(s):
> FileStreamSource (org.apache.kafka.connect.file.FileStreamSourceConnector)
> @ classpath
> FileStreamSink(org.apache.kafka.connect.file.FileStreamSinkConnector)
> @ classpath
> JdbcSource(io.confluent.connect.jdbc.JdbcSourceConnector)
> @ /usr/local/share/java/kafka-connect-jdbc
> MySql (io.debezium.connector.mysql.MySqlConnector)
> @ /usr/local/plugins/debezium-connector-mysql
> Converter(s):
> JsonConverter (org.apache.kafka.connect.json.JsonConverter)
> @ classpath
> ByteArrayConverter
> (org.apache.kafka.connect.converters.ByteArrayConverter)@ classpath
> SimpleHeaderConverter
> (org.apache.kafka.connect.converters.SimpleHeaderConverter) @ classpath
> AvroConverter (io.confluent.connect.avro.AvroConverter)
> @ /usr/local/share/java/kafka-serde-tools
> Transformation(s):
> InsertField (org.apache.kafka.connect.transforms.InsertField)
> @ classpath
> ReplaceField (org.apache.kafka.connect.transforms.ReplaceField)
> @ classpath
> MaskField (org.apache.kafka.connect.transforms.MaskField)
> @ classpath
> ValueToKey(org.apache.kafka.connect.transforms.ValueToKey)
> @ classpath
> HoistField(org.apache.kafka.connect.transforms.HoistField)
> @ classpath
> ExtractField (org.apache.kafka.connect.transforms.ExtractField)
> @ classpath
> SetSchemaMetadata (org.apache.kafka.connect.transforms.SetSchemaMetadata)
> @ classpath
> RegexRouter (org.apache.kafka.connect.transforms.RegexRouter)
> @ classpath
> TimestampRouter (org.apache.kafka.connect.transforms.TimestampRouter)
> @ classpath
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7157) Connect TimestampConverter SMT doesn't handle null values
[
https://issues.apache.org/jira/browse/KAFKA-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818827#comment-16818827
]
Valeria Vasylieva commented on KAFKA-7157:
--
[~rhauch] kindly ask you to look at the PR, thank you!
> Connect TimestampConverter SMT doesn't handle null values
> -
>
> Key: KAFKA-7157
> URL: https://issues.apache.org/jira/browse/KAFKA-7157
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
>
> TimestampConverter SMT is not able to handle null values (in any versions),
> so it's always trying to apply the transformation to the value. Instead, it
> needs to check for null and use the default value for the new schema's field.
> {noformat}
> [2018-07-03 02:31:52,490] ERROR Task MySourceConnector-2 threw an uncaught
> and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.transforms.TimestampConverter$2.toRaw(TimestampConverter.java:137)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.convertTimestamp(TimestampConverter.java:440)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.applyValueWithSchema(TimestampConverter.java:368)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.applyWithSchema(TimestampConverter.java:358)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.apply(TimestampConverter.java:275)
>
> at
> org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:38)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:435)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:264)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:182)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:150)
>
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> [2018-07-03 02:31:52,491] ERROR Task is being killed and will not recover
> until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765847#comment-16765847
]
Valeria Vasylieva commented on KAFKA-6755:
--
Hi [~rhauch], if it still needed to be implemented, I will work on it, ok?
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valeria Vasylieva reassigned KAFKA-6755:
Assignee: Valeria Vasylieva
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765976#comment-16765976
]
Valeria Vasylieva commented on KAFKA-6755:
--
[~rhauch] I am new to Kafka, could you please tell me, which should go first:
KIP or pull request?
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771800#comment-16771800
]
Valeria Vasylieva commented on KAFKA-6755:
--
hi [~rhauch]! I have submitted the PR, could you please look at it?
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
