> On Oct 18, 2017, at 7:05 AM, Jehan Bruggeman <jehanbrugge...@gmail.com> wrote: > > Hi all, > > quick follow-up: thank you Philip, you were right ! Indeed, I got rid of > this error by putting the converter in the connector's folder. > > I also tried something else: create a custom connector and use that custom > connector with the custom converter, both loaded as plugins. It also works. > > Summary: converters are loaded by the connector. If your connector is a > plugin, your converter should be as well. If you connector is not a plugin > (bundled with your kafka connect distrib), your converter should not be > either.
Yes, this strange behavior is due to the bug https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-6007. Once that is fixed, then this will not be a constraint. > > Jehan > > On 17 October 2017 at 14:51, Jehan Bruggeman <jehanbrugge...@gmail.com> > wrote: > >> HI Philip, >> >> thanks for the reply ! >> >> This issue looks like it might match my problem indeed. I'll look into it. >> >> Jehan >> >> On 16 October 2017 at 20:53, Philip Schmitt <philip.schm...@outlook.com> >> wrote: >> >>> Hi Jehan, >>> >>> I've run into the same issue last week and also got a "class could not be >>> found" error. >>> >>> Konstantine Karantasis helpfully pointed me towards >>> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-6007 >>> >>> To test this, I simply copied my SMT jar to the folder of the connector I >>> was using and adjusted the plugin.path property. >>> >>> I haven't fully tested tested it, but in my very quick initial test, it >>> seemed to have move past that "class could not be found" issue. >>> >>> Regards, >>> Philip >>> ________________________________ >>> From: jeh...@gmail.com <jeh...@gmail.com> on behalf of Jehan Bruggeman < >>> jehanbrugge...@gmail.com> >>> Sent: Monday, October 16, 2017 8:17 AM >>> To: users@kafka.apache.org >>> Subject: Re: Custom converter with Kafka Connect ? >>> >>> Hi Randall, >>> >>> thanks for your reply. I'm not sure about this; what detail could I add >>> that would help you figure it out ? >>> >>> Concerning the classpath: as described in my original email, I'm pretty >>> sure the jars are correctly added to the classpath since the classes in >>> the >>> jar are recognized by Kafka Connect when it starts (they are mentionned in >>> the logs, at least). >>> >>> (It's easier to read here, where I asked the same question: >>> https://stackoverflow.com/questions/46712095/using-a-custom- >>> converter-with-kafka-connect >>> ) >>> >>> thanks for your help ! >>> >>> Jehan >>> >>>> On 13 October 2017 at 16:07, Randall Hauch <rha...@gmail.com> wrote: >>>> >>>> On Tue, Oct 10, 2017 at 8:31 AM, Jehan Bruggeman < >>> jehanbrugge...@gmail.com >>>>> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> I'm trying to use a custom converter with Kafka Connect and I cannot >>> seem >>>>> to get it right. I'm hoping someone has experience with this and could >>>> help >>>>> me figure it out ! >>>>> >>>>> >>>>> Initial situation >>>>> ================ >>>>> >>>>> - my custom converter's class path is 'custom.CustomStringConverter'. >>>>> >>>>> - to avoid any mistakes, my custom converter is currently just a >>>> copy/paste >>>>> of the pre-existing StringConverter (of course, this will change when >>>> I'll >>>>> get it to work). >>>>> https://github.com/apache/kafka/blob/trunk/connect/api/ >>>>> src/main/java/org/apache/kafka/connect/storage/StringConverter.java >>>>> >>>>> - I have a kafka connect cluster of 3 nodes, The nodes are running >>>>> confluent's official docker images ( confluentinc/cp-kafka-connect: >>> 3.3.0 >>>>> ). >>>>> >>>>> - Each node is configured to load a jar with my converter in it >>> (using a >>>>> docker volume). >>>>> >>>> >>>> Can you explain this in more detail? Make sure that you add the JAR to >>> the >>>> classpath. >>>> >>>> >>>>> >>>>> >>>>> >>>>> What happens ? >>>>> ================ >>>>> >>>>> When the connectors start, they correctly load the jars and find the >>>> custom >>>>> converter. Indeed, this is what I see in the logs : >>>>> >>>>> [2017-10-10 13:06:46,274] INFO Registered loader: >>>>> PluginClassLoader{pluginLocation=file:/opt/custom-connectors/custom- >>>>> converter-1.0-SNAPSHOT.jar} >>>>> (org.apache.kafka.connect.runtime.isolation.DelegatingClassL >>> oader:199) >>>>> [2017-10-10 13:06:46,274] INFO Added plugin >>>> 'custom.CustomStringConverter' >>>>> (org.apache.kafka.connect.runtime.isolation.DelegatingClassL >>> oader:132) >>>>> [...] >>>>> [2017-10-10 13:07:43,454] INFO Added aliases 'CustomStringConverter' >>> and >>>>> 'CustomString' to plugin 'custom.CustomStringConverter' >>>>> (org.apache.kafka.connect.runtime.isolation.DelegatingClassL >>> oader:293) >>>>> >>>>> I then POST a JSON config to one of the connector nodes to create my >>>>> connector : >>>>> >>>>> { >>>>> "name": "hdfsSinkCustom", >>>>> "config": { >>>>> "topics": "yellow", >>>>> "tasks.max": "1", >>>>> "key.converter": "org.apache.kafka.connect.stor >>> age.StringConverter", >>>>> "value.converter": "custom.CustomStringConverter", >>>>> "connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector", >>>>> "hdfs.url": "hdfs://hdfs-namenode:8020/hdfs-sink", >>>>> "topics.dir": "yellow_storage", >>>>> "flush.size": "1", >>>>> "rotate.interval.ms": "1000" >>>>> } >>>>> } >>>>> >>>>> And receive the following reply : >>>>> >>>>> { >>>>> "error_code": 400, >>>>> "message": "Connector configuration is invalid and contains the >>>>> following 1 error(s):\nInvalid value custom.CustomStringConverter for >>>>> configuration value.converter: Class custom.CustomStringConverter >>> could >>>> not >>>>> be found.\nYou can also find the above list of errors at the endpoint >>>>> `/{connectorType}/config/validate`" >>>>> } >>>>> >>>>> ================ >>>>> >>>>> If I try running Kafka Connect stadnalone, the error message is the >>> same. >>>>> >>>>> Has anybody faced this already ? What am I missing ? >>>>> >>>>> Many thanks to anybody reading this ! >>>>> >>>>> Jehan >>>>> >>>> >>> >> >>
