Re: Creating a connector with Kafka Connect Distributed returning 500 error

[Phillip Mann]

Hello Konstantine and community,

I was able to fix this problem by using the latest version of Confluent 
Platform.  I was running CP 3.0.1 but upgraded to 3.1.1 and my worker and 
connector behaved as expected.  Thanks!

Phillip

From: Konstantine Karantasis <konstant...@confluent.io>
Date: Wednesday, December 7, 2016 at 1:02 PM
To: Phillip Mann <pm...@trulia.com>
Cc: "users@kafka.apache.org" <users@kafka.apache.org>
Subject: Re: Creating a connector with Kafka Connect Distributed returning 500 
error


The bug I was referring to was only in trunk for just a while. Thus, your issue 
must be related to something else, even though the response statuses are 
similar.

Let me know if you want to share a bigger and more detailed (DEBUG level at 
least) snapshot of the parts of the logs that might be related to this failure.

Cheers,
Konstantine

On Wed, Dec 7, 2016 at 11:15 AM, Phillip Mann 
<pm...@trulia.com<mailto:pm...@trulia.com>> wrote:
Hello Konstantine,

Thanks for your reply.

I am using Confluent 3.0.1 installed on my machine and our cluster.  However, 
our AWS cluster has Confluent 3.1.1 installed so I will test with 3.1.1 client 
and cluster and see if this resolves the issue.  Additionally, I’ll use the 
debug levels if this does not resolve my issue.

If not, I’ll explore the trunk repo but I would prefer to use stable versions 
of CP / Kafka that can be accessed with Maven.

Thanks again.

Phillip

> Hi Phillip,
>
> may I ask which Kafka version did you use?
>
> trunk repo in Apache Kafka contained briefly a bug in Connect framework
> (during the past week) that produced failures similar to the one you
> describe (only in distributed mode). A fix has been pushed since yesterday.
>
> 3) Some useful step-by-step information is provided in the quickstart guide
> here:
> https://kafka.apache.org/quickstart#quickstart_kafkaconnect
>
> as well as in the documentation of Confluent:
> http://docs.confluent.io/3.1.0/connect/quickstart.html#<http://docs.confluent.io/3.1.0/connect/quickstart.html>
>
> Alternatively, you might want to follow the quickstart guide of one of the
> open source connectors, here:
> http://docs.confluent.io/3.1.0/connect/connectors.html
>
> 2) From what you mention above, it seems more like that you're hitting this
> temporary bug. But again that depends on which Kafka version you've been
> using.
>
> 1) Generating logs, in one of the debug levels (e.g. DEBUG, TRACE) is
> usually a useful source of information.
> Alternatively you may chose to run Connect in debug mode by setting the
> environment variable KAFKA_DEBUG and attaching a remote debugger to it
> (such as IntelliJ's remote debugging capability). With respect to live
> debugging, we are planning to post a step-by-step guide for Kafka and Kafka
> Connect soon.
>
> Regards,
> Konstantine
>
>> On Tue, Dec 6, 2016 at 11:22 AM, Phillip Mann 
>> <pm...@trulia.com<mailto:pm...@trulia.com>> wrote:
>>
>> I am working on migrating from Camus to Kafka Connect. I am working on the
>> implementation of Kafka Connect and specifically focused on distributed
>> mode. I am able to start a worker successfully on my local machine which I
>> assume communicates with my Kafka cluster. I am further able to run two GET
>> commands such as / and /connector-plugins which return the correct JSON.
>> However, when I try to POST a command to create a connector, I receive a
>> 500 error and a time out. Specifically, I use this command to POST for
>> testing:
>>
>> curl -X POST -H "Content-Type: application/json" --data '{"name":
>> "local-file-sink", "config": {"connector.class":"FileStreamSinkConnector",
>> "tasks.max":"1", "file":"test.sink.txt", "topics":"myTopic" }}'
>> localhost:8083/connectors
>>
>> and eventually I get this response:
>>
>> {"error_code": 500, "message": "Request timed out"}
>>
>> I am lost as to what is going on. The logs from my Kafka Connect
>> distributed worker show this:
>>
>> [2016-12-05 14:34:32,436] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:22:34:32
>> +] "GET /connector-plugins HTTP/1.1" 200 315  2
>> (org.apache.kafka.connect.runtime.rest.RestServer:60)
>> [2016-12-05 15:05:25,422] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:25
>> +] "GET /connector-plugins HTTP/1.1" 200 315  3
>> (org.apache.kafka.connect.runtime.rest.RestServer:60)
>> [2016-12-05 15:05:28,389] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:28
>> +] "GET /connector-plugins HTTP/1.1" 200 315  2
>> (org.apache.kafk

Re: Creating a connector with Kafka Connect Distributed returning 500 error

[Konstantine Karantasis]

The bug I was referring to was only in trunk for just a while. Thus, your
issue must be related to something else, even though the response statuses
are similar.

Let me know if you want to share a bigger and more detailed (DEBUG level at
least) snapshot of the parts of the logs that might be related to this
failure.

Cheers,
Konstantine

On Wed, Dec 7, 2016 at 11:15 AM, Phillip Mann  wrote:

> Hello Konstantine,
>
>
>
> Thanks for your reply.
>
>
>
> I am using Confluent 3.0.1 installed on my machine and our cluster.
> However, our AWS cluster has Confluent 3.1.1 installed so I will test with
> 3.1.1 client and cluster and see if this resolves the issue.  Additionally,
> I’ll use the debug levels if this does not resolve my issue.
>
>
>
> If not, I’ll explore the trunk repo but I would prefer to use stable
> versions of CP / Kafka that can be accessed with Maven.
>
>
>
> Thanks again.
>
>
>
> Phillip
>
>
>
> > Hi Phillip,
>
> >
>
> > may I ask which Kafka version did you use?
>
> >
>
> > trunk repo in Apache Kafka contained briefly a bug in Connect framework
>
> > (during the past week) that produced failures similar to the one you
>
> > describe (only in distributed mode). A fix has been pushed since
> yesterday.
>
> >
>
> > 3) Some useful step-by-step information is provided in the quickstart
> guide
>
> > here:
>
> > https://kafka.apache.org/quickstart#quickstart_kafkaconnect
>
> >
>
> > as well as in the documentation of Confluent:
>
> > http://docs.confluent.io/3.1.0/connect/quickstart.html#
>
> >
>
> > Alternatively, you might want to follow the quickstart guide of one of
> the
>
> > open source connectors, here:
>
> > http://docs.confluent.io/3.1.0/connect/connectors.html
>
> >
>
> > 2) From what you mention above, it seems more like that you're hitting
> this
>
> > temporary bug. But again that depends on which Kafka version you've been
>
> > using.
>
> >
>
> > 1) Generating logs, in one of the debug levels (e.g. DEBUG, TRACE) is
>
> > usually a useful source of information.
>
> > Alternatively you may chose to run Connect in debug mode by setting the
>
> > environment variable KAFKA_DEBUG and attaching a remote debugger to it
>
> > (such as IntelliJ's remote debugging capability). With respect to live
>
> > debugging, we are planning to post a step-by-step guide for Kafka and
> Kafka
>
> > Connect soon.
>
> >
>
> > Regards,
>
> > Konstantine
>
> >
>
> >> On Tue, Dec 6, 2016 at 11:22 AM, Phillip Mann  wrote:
>
> >>
>
> >> I am working on migrating from Camus to Kafka Connect. I am working on
> the
>
> >> implementation of Kafka Connect and specifically focused on distributed
>
> >> mode. I am able to start a worker successfully on my local machine
> which I
>
> >> assume communicates with my Kafka cluster. I am further able to run two
> GET
>
> >> commands such as / and /connector-plugins which return the correct JSON.
>
> >> However, when I try to POST a command to create a connector, I receive a
>
> >> 500 error and a time out. Specifically, I use this command to POST for
>
> >> testing:
>
> >>
>
> >> curl -X POST -H "Content-Type: application/json" --data '{"name":
>
> >> "local-file-sink", "config": {"connector.class":"
> FileStreamSinkConnector",
>
> >> "tasks.max":"1", "file":"test.sink.txt", "topics":"myTopic" }}'
>
> >> localhost:8083/connectors
>
> >>
>
> >> and eventually I get this response:
>
> >>
>
> >> {"error_code": 500, "message": "Request timed out"}
>
> >>
>
> >> I am lost as to what is going on. The logs from my Kafka Connect
>
> >> distributed worker show this:
>
> >>
>
> >> [2016-12-05 14:34:32,436] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:22:34:32
>
> >> +] "GET /connector-plugins HTTP/1.1" 200 315  2
>
> >> (org.apache.kafka.connect.runtime.rest.RestServer:60)
>
> >> [2016-12-05 15:05:25,422] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:25
>
> >> +] "GET /connector-plugins HTTP/1.1" 200 315  3
>
> >> (org.apache.kafka.connect.runtime.rest.RestServer:60)
>
> >> [2016-12-05 15:05:28,389] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:28
>
> >> +] "GET /connector-plugins HTTP/1.1" 200 315  2
>
> >> (org.apache.kafka.connect.runtime.rest.RestServer:60)
>
> >> [2016-12-05 15:07:38,644] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:06:08
>
> >> +] "GET /connectors HTTP/1.1" 500 48  90003
> (org.apache.kafka.connect.
>
> >> runtime.rest.RestServer:60)
>
> >> [2016-12-05 15:07:44,450] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:07:44
>
> >> +] "GET /connector-plugins HTTP/1.1" 200 315  1
>
> >> (org.apache.kafka.connect.runtime.rest.RestServer:60)
>
> >> [2016-12-05 15:13:06,703] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:11:36
>
> >> +] "POST /connectors HTTP/1.1" 500 48  90003
> (org.apache.kafka.connect.
>
> >> runtime.rest.RestServer:60)
>
> >> [2016-12-05 15:15:38,506] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:14:08
>
> >> +] "POST /connectors HTTP/1.1" 500 48  90005
> (org.apache.kafka.connect.
>
> >> runtime.rest.RestServer:60)
>

Re: Creating a connector with Kafka Connect Distributed returning 500 error

[Phillip Mann]

Hello Konstantine,

Thanks for your reply.

I am using Confluent 3.0.1 installed on my machine and our cluster.  However, 
our AWS cluster has Confluent 3.1.1 installed so I will test with 3.1.1 client 
and cluster and see if this resolves the issue.  Additionally, I’ll use the 
debug levels if this does not resolve my issue.

If not, I’ll explore the trunk repo but I would prefer to use stable versions 
of CP / Kafka that can be accessed with Maven.

Thanks again.

Phillip

> Hi Phillip,
>
> may I ask which Kafka version did you use?
>
> trunk repo in Apache Kafka contained briefly a bug in Connect framework
> (during the past week) that produced failures similar to the one you
> describe (only in distributed mode). A fix has been pushed since yesterday.
>
> 3) Some useful step-by-step information is provided in the quickstart guide
> here:
> https://kafka.apache.org/quickstart#quickstart_kafkaconnect
>
> as well as in the documentation of Confluent:
> http://docs.confluent.io/3.1.0/connect/quickstart.html#
>
> Alternatively, you might want to follow the quickstart guide of one of the
> open source connectors, here:
> http://docs.confluent.io/3.1.0/connect/connectors.html
>
> 2) From what you mention above, it seems more like that you're hitting this
> temporary bug. But again that depends on which Kafka version you've been
> using.
>
> 1) Generating logs, in one of the debug levels (e.g. DEBUG, TRACE) is
> usually a useful source of information.
> Alternatively you may chose to run Connect in debug mode by setting the
> environment variable KAFKA_DEBUG and attaching a remote debugger to it
> (such as IntelliJ's remote debugging capability). With respect to live
> debugging, we are planning to post a step-by-step guide for Kafka and Kafka
> Connect soon.
>
> Regards,
> Konstantine
>
>> On Tue, Dec 6, 2016 at 11:22 AM, Phillip Mann  wrote:
>>
>> I am working on migrating from Camus to Kafka Connect. I am working on the
>> implementation of Kafka Connect and specifically focused on distributed
>> mode. I am able to start a worker successfully on my local machine which I
>> assume communicates with my Kafka cluster. I am further able to run two GET
>> commands such as / and /connector-plugins which return the correct JSON.
>> However, when I try to POST a command to create a connector, I receive a
>> 500 error and a time out. Specifically, I use this command to POST for
>> testing:
>>
>> curl -X POST -H "Content-Type: application/json" --data '{"name":
>> "local-file-sink", "config": {"connector.class":"FileStreamSinkConnector",
>> "tasks.max":"1", "file":"test.sink.txt", "topics":"myTopic" }}'
>> localhost:8083/connectors
>>
>> and eventually I get this response:
>>
>> {"error_code": 500, "message": "Request timed out"}
>>
>> I am lost as to what is going on. The logs from my Kafka Connect
>> distributed worker show this:
>>
>> [2016-12-05 14:34:32,436] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:22:34:32
>> +] "GET /connector-plugins HTTP/1.1" 200 315  2
>> (org.apache.kafka.connect.runtime.rest.RestServer:60)
>> [2016-12-05 15:05:25,422] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:25
>> +] "GET /connector-plugins HTTP/1.1" 200 315  3
>> (org.apache.kafka.connect.runtime.rest.RestServer:60)
>> [2016-12-05 15:05:28,389] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:28
>> +] "GET /connector-plugins HTTP/1.1" 200 315  2
>> (org.apache.kafka.connect.runtime.rest.RestServer:60)
>> [2016-12-05 15:07:38,644] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:06:08
>> +] "GET /connectors HTTP/1.1" 500 48  90003 (org.apache.kafka.connect.
>> runtime.rest.RestServer:60)
>> [2016-12-05 15:07:44,450] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:07:44
>> +] "GET /connector-plugins HTTP/1.1" 200 315  1
>> (org.apache.kafka.connect.runtime.rest.RestServer:60)
>> [2016-12-05 15:13:06,703] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:11:36
>> +] "POST /connectors HTTP/1.1" 500 48  90003 (org.apache.kafka.connect.
>> runtime.rest.RestServer:60)
>> [2016-12-05 15:15:38,506] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:14:08
>> +] "POST /connectors HTTP/1.1" 500 48  90005 (org.apache.kafka.connect.
>> runtime.rest.RestServer:60)
>>
>> where you can see the error codes and the commands.
>>
>> I guess my main questions and issues are:
>>
>>   1.  How can I better debug Kafka Connect so I can try and fix this?
>>   2.  Is there anything that I'm doing that is glaringly wrong?
>>   3.  Is there any step-by-step documentation or blog posts on getting a
>> Kafka Connect distributed worker and connector to run? I have not really
>> seen anything or even best practices kinds of documentation? Maybe I am
>> just way too early of an adopter.
>>
>> I look forward to hearing back from the community and thank you for your
>> help!
>>
>>

Re: Creating a connector with Kafka Connect Distributed returning 500 error

[Konstantine Karantasis]

Hi Phillip,

may I ask which Kafka version did you use?

trunk repo in Apache Kafka contained briefly a bug in Connect framework
(during the past week) that produced failures similar to the one you
describe (only in distributed mode). A fix has been pushed since yesterday.

3) Some useful step-by-step information is provided in the quickstart guide
here:
https://kafka.apache.org/quickstart#quickstart_kafkaconnect

as well as in the documentation of Confluent:
http://docs.confluent.io/3.1.0/connect/quickstart.html#

Alternatively, you might want to follow the quickstart guide of one of the
open source connectors, here:
http://docs.confluent.io/3.1.0/connect/connectors.html

2) From what you mention above, it seems more like that you're hitting this
temporary bug. But again that depends on which Kafka version you've been
using.

1) Generating logs, in one of the debug levels (e.g. DEBUG, TRACE) is
usually a useful source of information.
Alternatively you may chose to run Connect in debug mode by setting the
environment variable KAFKA_DEBUG and attaching a remote debugger to it
(such as IntelliJ's remote debugging capability). With respect to live
debugging, we are planning to post a step-by-step guide for Kafka and Kafka
Connect soon.

Regards,
Konstantine

On Tue, Dec 6, 2016 at 11:22 AM, Phillip Mann  wrote:

> I am working on migrating from Camus to Kafka Connect. I am working on the
> implementation of Kafka Connect and specifically focused on distributed
> mode. I am able to start a worker successfully on my local machine which I
> assume communicates with my Kafka cluster. I am further able to run two GET
> commands such as / and /connector-plugins which return the correct JSON.
> However, when I try to POST a command to create a connector, I receive a
> 500 error and a time out. Specifically, I use this command to POST for
> testing:
>
> curl -X POST -H "Content-Type: application/json" --data '{"name":
> "local-file-sink", "config": {"connector.class":"FileStreamSinkConnector",
> "tasks.max":"1", "file":"test.sink.txt", "topics":"myTopic" }}'
> localhost:8083/connectors
>
> and eventually I get this response:
>
> {"error_code": 500, "message": "Request timed out"}
>
> I am lost as to what is going on. The logs from my Kafka Connect
> distributed worker show this:
>
> [2016-12-05 14:34:32,436] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:22:34:32
> +] "GET /connector-plugins HTTP/1.1" 200 315  2
> (org.apache.kafka.connect.runtime.rest.RestServer:60)
> [2016-12-05 15:05:25,422] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:25
> +] "GET /connector-plugins HTTP/1.1" 200 315  3
> (org.apache.kafka.connect.runtime.rest.RestServer:60)
> [2016-12-05 15:05:28,389] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:28
> +] "GET /connector-plugins HTTP/1.1" 200 315  2
> (org.apache.kafka.connect.runtime.rest.RestServer:60)
> [2016-12-05 15:07:38,644] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:06:08
> +] "GET /connectors HTTP/1.1" 500 48  90003 (org.apache.kafka.connect.
> runtime.rest.RestServer:60)
> [2016-12-05 15:07:44,450] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:07:44
> +] "GET /connector-plugins HTTP/1.1" 200 315  1
> (org.apache.kafka.connect.runtime.rest.RestServer:60)
> [2016-12-05 15:13:06,703] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:11:36
> +] "POST /connectors HTTP/1.1" 500 48  90003 (org.apache.kafka.connect.
> runtime.rest.RestServer:60)
> [2016-12-05 15:15:38,506] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:14:08
> +] "POST /connectors HTTP/1.1" 500 48  90005 (org.apache.kafka.connect.
> runtime.rest.RestServer:60)
>
> where you can see the error codes and the commands.
>
> I guess my main questions and issues are:
>
>   1.  How can I better debug Kafka Connect so I can try and fix this?
>   2.  Is there anything that I'm doing that is glaringly wrong?
>   3.  Is there any step-by-step documentation or blog posts on getting a
> Kafka Connect distributed worker and connector to run? I have not really
> seen anything or even best practices kinds of documentation? Maybe I am
> just way too early of an adopter.
>
> I look forward to hearing back from the community and thank you for your
> help!
>
>

Creating a connector with Kafka Connect Distributed returning 500 error

[Phillip Mann]

I am working on migrating from Camus to Kafka Connect. I am working on the 
implementation of Kafka Connect and specifically focused on distributed mode. I 
am able to start a worker successfully on my local machine which I assume 
communicates with my Kafka cluster. I am further able to run two GET commands 
such as / and /connector-plugins which return the correct JSON. However, when I 
try to POST a command to create a connector, I receive a 500 error and a time 
out. Specifically, I use this command to POST for testing:

curl -X POST -H "Content-Type: application/json" --data '{"name": 
"local-file-sink", "config": {"connector.class":"FileStreamSinkConnector", 
"tasks.max":"1", "file":"test.sink.txt", "topics":"myTopic" }}' 
localhost:8083/connectors

and eventually I get this response:

{"error_code": 500, "message": "Request timed out"}

I am lost as to what is going on. The logs from my Kafka Connect distributed 
worker show this:

[2016-12-05 14:34:32,436] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:22:34:32 +] 
"GET /connector-plugins HTTP/1.1" 200 315  2 
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:05:25,422] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:25 +] 
"GET /connector-plugins HTTP/1.1" 200 315  3 
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:05:28,389] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:28 +] 
"GET /connector-plugins HTTP/1.1" 200 315  2 
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:07:38,644] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:06:08 +] 
"GET /connectors HTTP/1.1" 500 48  90003 
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:07:44,450] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:07:44 +] 
"GET /connector-plugins HTTP/1.1" 200 315  1 
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:13:06,703] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:11:36 +] 
"POST /connectors HTTP/1.1" 500 48  90003 
(org.apache.kafka.connect.runtime.rest.RestServer:60)
[2016-12-05 15:15:38,506] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:14:08 +] 
"POST /connectors HTTP/1.1" 500 48  90005 
(org.apache.kafka.connect.runtime.rest.RestServer:60)

where you can see the error codes and the commands.

I guess my main questions and issues are:

  1.  How can I better debug Kafka Connect so I can try and fix this?
  2.  Is there anything that I'm doing that is glaringly wrong?
  3.  Is there any step-by-step documentation or blog posts on getting a Kafka 
Connect distributed worker and connector to run? I have not really seen 
anything or even best practices kinds of documentation? Maybe I am just way too 
early of an adopter.

I look forward to hearing back from the community and thank you for your help!