[
https://issues.apache.org/jira/browse/KAFKA-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336254#comment-15336254
]
Randall Hauch commented on KAFKA-3487:
--------------------------------------
I hope this can move toward an approach that can eventually support the Jigsaw
module system that appears to be coming in JDK9. Jigsaw relies upon the
standard Java service loader mechanism (added in JDK6) to find all
implementations of an interface, and Jigsaw will then properly handle all
dependencies and class loading.
A pre-JDK9 approach is to use a configuration parameter for the workers (e.g.,
a connector module path) that simply lists the directories in which Kafka
Connect can find each connector "module". The connector module's directory
would contain all of the JARs required by the connector. Upon startup, Kafka
Connect could iterate through this list of paths, and for each: create a URL
classloader (that inherits the parent classpath), pass the URL classloader to
the service loader method to load the connector implementation class (without
having to know its name), create a {{ConnectorModule}} object with the URL
classloader and reference to the connector class, and then load the
{{ConnectorModule}} into an internal registry keyed by name. Then, the rest of
Kafka Connect would simply use the registry.
Really, this same registry could also be used to find all implementations
available on the current classloader, meaning it would work with connectors
that don't define a
{{META-INF/services/org.apache.kafka.connect.connector.Connector}} file for the
service loader.
And, to add support for JDK9, upon startup Kafka Connect would simply use the
service loader to locate all {{Connector}} implementation classes, and populate
the same registry, using a different {{ConnectorModule}} implementation that
simply instantiates the class and relying upon JDK9 modules to properly handle
all of the classloading and isolation. The rest of the Kafka Connect remains
unchanged.
The only change to connector implementations is to add support for the service
loader, and since that's going to be required by JDK9 it might be worth doing
now. And, doing this now would greatly simply the implementation.
> Support per-connector/per-task classloaders in Connect
> ------------------------------------------------------
>
> Key: KAFKA-3487
> URL: https://issues.apache.org/jira/browse/KAFKA-3487
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect
> Affects Versions: 0.10.0.0
> Reporter: Ewen Cheslack-Postava
> Assignee: Liquan Pei
> Priority: Critical
> Labels: needs-kip
>
> Currently we just use the default ClassLoader in Connect. However, this
> limits how we can compatibly load conflicting connector plugins. Ideally we
> would use a separate class loader per connector/task that is instantiated to
> avoid potential conflicts.
> Note that this also opens up options for other ways to provide jars to
> instantiate connectors. For example, Spark uses this to dynamically publish
> classes defined in the REPL and load them via URL:
> https://ardoris.wordpress.com/2014/03/30/how-spark-does-class-loading/ But
> much simpler examples (include URL in the connector class instead of just
> class name) are also possible and could be a nice way to more support dynamic
> sets of connectors, multiple versions of the same connector, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
