Hi Phillip, may I ask which Kafka version did you use?
trunk repo in Apache Kafka contained briefly a bug in Connect framework (during the past week) that produced failures similar to the one you describe (only in distributed mode). A fix has been pushed since yesterday. 3) Some useful step-by-step information is provided in the quickstart guide here: https://kafka.apache.org/quickstart#quickstart_kafkaconnect as well as in the documentation of Confluent: http://docs.confluent.io/3.1.0/connect/quickstart.html# Alternatively, you might want to follow the quickstart guide of one of the open source connectors, here: http://docs.confluent.io/3.1.0/connect/connectors.html 2) From what you mention above, it seems more like that you're hitting this temporary bug. But again that depends on which Kafka version you've been using. 1) Generating logs, in one of the debug levels (e.g. DEBUG, TRACE) is usually a useful source of information. Alternatively you may chose to run Connect in debug mode by setting the environment variable KAFKA_DEBUG and attaching a remote debugger to it (such as IntelliJ's remote debugging capability). With respect to live debugging, we are planning to post a step-by-step guide for Kafka and Kafka Connect soon. Regards, Konstantine On Tue, Dec 6, 2016 at 11:22 AM, Phillip Mann <pm...@trulia.com> wrote: > I am working on migrating from Camus to Kafka Connect. I am working on the > implementation of Kafka Connect and specifically focused on distributed > mode. I am able to start a worker successfully on my local machine which I > assume communicates with my Kafka cluster. I am further able to run two GET > commands such as / and /connector-plugins which return the correct JSON. > However, when I try to POST a command to create a connector, I receive a > 500 error and a time out. Specifically, I use this command to POST for > testing: > > curl -X POST -H "Content-Type: application/json" --data '{"name": > "local-file-sink", "config": {"connector.class":"FileStreamSinkConnector", > "tasks.max":"1", "file":"test.sink.txt", "topics":"myTopic" }}' > localhost:8083/connectors > > and eventually I get this response: > > {"error_code": 500, "message": "Request timed out"} > > I am lost as to what is going on. The logs from my Kafka Connect > distributed worker show this: > > [2016-12-05 14:34:32,436] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:22:34:32 > +0000] "GET /connector-plugins HTTP/1.1" 200 315 2 > (org.apache.kafka.connect.runtime.rest.RestServer:60) > [2016-12-05 15:05:25,422] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:25 > +0000] "GET /connector-plugins HTTP/1.1" 200 315 3 > (org.apache.kafka.connect.runtime.rest.RestServer:60) > [2016-12-05 15:05:28,389] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:28 > +0000] "GET /connector-plugins HTTP/1.1" 200 315 2 > (org.apache.kafka.connect.runtime.rest.RestServer:60) > [2016-12-05 15:07:38,644] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:06:08 > +0000] "GET /connectors HTTP/1.1" 500 48 90003 (org.apache.kafka.connect. > runtime.rest.RestServer:60) > [2016-12-05 15:07:44,450] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:07:44 > +0000] "GET /connector-plugins HTTP/1.1" 200 315 1 > (org.apache.kafka.connect.runtime.rest.RestServer:60) > [2016-12-05 15:13:06,703] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:11:36 > +0000] "POST /connectors HTTP/1.1" 500 48 90003 (org.apache.kafka.connect. > runtime.rest.RestServer:60) > [2016-12-05 15:15:38,506] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:14:08 > +0000] "POST /connectors HTTP/1.1" 500 48 90005 (org.apache.kafka.connect. > runtime.rest.RestServer:60) > > where you can see the error codes and the commands. > > I guess my main questions and issues are: > > 1. How can I better debug Kafka Connect so I can try and fix this? > 2. Is there anything that I'm doing that is glaringly wrong? > 3. Is there any step-by-step documentation or blog posts on getting a > Kafka Connect distributed worker and connector to run? I have not really > seen anything or even best practices kinds of documentation? Maybe I am > just way too early of an adopter. > > I look forward to hearing back from the community and thank you for your > help! > >
