[ https://issues.apache.org/jira/browse/KAFKA-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199252#comment-16199252 ]
Konstantine Karantasis commented on KAFKA-6007: ----------------------------------------------- My explanation so far is as follows: Normally, when a connector's classloader can't find another plugin within its path (the classloader's set of urls), it will delegate class loading to its parent. This works for all the plugin classloaders that are created for plugins residing in the {{plugin.path}}. Upon such delegation, the delegating classloader, which is the parent of every plugin classloader, will be able to elect the appropriate plugin classloader for the plugin that is nested in the connector's configuration. However, if the connector is residing in the system's classpath and not in any of the {{plugin.path}} locations, this means that its classloader is the system classloader. This classloader's parent cannot be overridden and set to be the delegating classloader. Thus, when a connector loaded from the system classpath, requires a transform or other plugin, this plugin needs also to reside in the system classpath (as opposed to the {{plugin.path}}). The solution would be to move all the plugins with interdependencies to each other, either to the {{plugin.path}} altogether or the system classpath again as a whole set. > Connect can't validate against transforms in plugins.path > --------------------------------------------------------- > > Key: KAFKA-6007 > URL: https://issues.apache.org/jira/browse/KAFKA-6007 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect > Affects Versions: 0.11.0.0 > Reporter: Stephane Maarek > Assignee: Konstantine Karantasis > Priority: Blocker > Fix For: 0.11.0.1, 1.0.0 > > > Kafka Connect can't validate a custom transformation if placed in plugins > path. > Here's the output I get on the validate call: > {code:java} > Invalid value com.mycorp.kafka.transforms.impl.FlattenSinkRecord for > configuration transforms.Flat.type: Class > com.mycorp.kafka.transforms.impl.FlattenSinkRecord could not be found. > Invalid value null for configuration transforms.Flat.type: Not a > Transformation > "recommended_values": [ > "com.mycorp.kafka.transforms.Flatten$Key", > "com.mycorp.kafka.transforms.Flatten$Value", > "com.mycorp.kafka.transforms.impl.FlattenSinkRecord", > "org.apache.kafka.connect.transforms.Cast$Key", > "org.apache.kafka.connect.transforms.Cast$Value", > "org.apache.kafka.connect.transforms.ExtractField$Key", > "org.apache.kafka.connect.transforms.ExtractField$Value", > "org.apache.kafka.connect.transforms.Flatten$Key", > "org.apache.kafka.connect.transforms.Flatten$Value", > "org.apache.kafka.connect.transforms.HoistField$Key", > "org.apache.kafka.connect.transforms.HoistField$Value", > "org.apache.kafka.connect.transforms.InsertField$Key", > "org.apache.kafka.connect.transforms.InsertField$Value", > "org.apache.kafka.connect.transforms.MaskField$Key", > "org.apache.kafka.connect.transforms.MaskField$Value", > "org.apache.kafka.connect.transforms.RegexRouter", > "org.apache.kafka.connect.transforms.ReplaceField$Key", > "org.apache.kafka.connect.transforms.ReplaceField$Value", > "org.apache.kafka.connect.transforms.SetSchemaMetadata$Key", > "org.apache.kafka.connect.transforms.SetSchemaMetadata$Value", > "org.apache.kafka.connect.transforms.TimestampConverter$Key", > "org.apache.kafka.connect.transforms.TimestampConverter$Value", > "org.apache.kafka.connect.transforms.TimestampRouter", > "org.apache.kafka.connect.transforms.ValueToKey"], > {code} > As you can see the class appear in the recommended values (!) but can't be > picked up on the validate call. > I believe it's because the recommender implements class discovery using > plugins: > https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectorConfig.java#L194 > But the class inference itself doesn't: > https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ConnectorConfig.java#L199 > (I'm not an expert in class loading though, just a guess... Unsure how to fix) > A quick fix is to add the transformations in the ClassPath itself, but that > defeats the point a bit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)