[jira] [Created] (KAFKA-4518) Kafka-connect Not starting after DB restarts
Akshath Patkar created KAFKA-4518:
-
Summary: Kafka-connect Not starting after DB restarts
Key: KAFKA-4518
URL: https://issues.apache.org/jira/browse/KAFKA-4518
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Akshath Patkar
Assignee: Ewen Cheslack-Postava
We use confluent3.0 package+Debezium for steaming mysql data into Kafka.
How ever we had some minor issues with mysql and had to restart mysql servers.
After the restart of mysql server, KC stops(this is expected, its there on DBZ
docs).
We restarted KC, later kaka-connect starts without any error, But it will not
stream the data into Kafka.
We can see following like in logs and connect Just sits there, without
committing any thing to kafka.
INFO Source task WorkerSourceTask{id=XX} finished initialization and start
(org.apache.kafka.connect.runtime.WorkerSourceTask:138)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (KAFKA-4518) Kafka-connect Not starting after DB restarts
[
https://issues.apache.org/jira/browse/KAFKA-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshath Patkar resolved KAFKA-4518.
---
Resolution: Not A Problem
> Kafka-connect Not starting after DB restarts
>
>
> Key: KAFKA-4518
> URL: https://issues.apache.org/jira/browse/KAFKA-4518
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Akshath Patkar
>Assignee: Ewen Cheslack-Postava
>
> We use confluent3.0 package+Debezium for steaming mysql data into Kafka.
> How ever we had some minor issues with mysql and had to restart mysql
> servers. After the restart of mysql server, KC stops(this is expected, its
> there on DBZ docs).
> We restarted KC, later kaka-connect starts without any error, But it will not
> stream the data into Kafka.
> We can see following like in logs and connect Just sits there, without
> committing any thing to kafka.
> INFO Source task WorkerSourceTask{id=XX} finished initialization and
> start (org.apache.kafka.connect.runtime.WorkerSourceTask:138)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-4518) Kafka-connect Not starting after DB restarts
[
https://issues.apache.org/jira/browse/KAFKA-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742761#comment-15742761
]
Akshath Patkar commented on KAFKA-4518:
---
This looks like a issue with server.id config on DBZ. We used same server.id
for 4 connectors.
corrected the config and looks like we are seeing commits from all connectors,
Closing this.
> Kafka-connect Not starting after DB restarts
>
>
> Key: KAFKA-4518
> URL: https://issues.apache.org/jira/browse/KAFKA-4518
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Akshath Patkar
>Assignee: Ewen Cheslack-Postava
>
> We use confluent3.0 package+Debezium for steaming mysql data into Kafka.
> How ever we had some minor issues with mysql and had to restart mysql
> servers. After the restart of mysql server, KC stops(this is expected, its
> there on DBZ docs).
> We restarted KC, later kaka-connect starts without any error, But it will not
> stream the data into Kafka.
> We can see following like in logs and connect Just sits there, without
> committing any thing to kafka.
> INFO Source task WorkerSourceTask{id=XX} finished initialization and
> start (org.apache.kafka.connect.runtime.WorkerSourceTask:138)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-3367) Delete topic dont delete the complete log from kafka
Akshath Patkar created KAFKA-3367: - Summary: Delete topic dont delete the complete log from kafka Key: KAFKA-3367 URL: https://issues.apache.org/jira/browse/KAFKA-3367 Project: Kafka Issue Type: Improvement Reporter: Akshath Patkar Delete topic Just marks the topic as deleted. But data still remain in logs. How can we delete the topic completely with out doing manual delete of logs from kafka and zookeeper -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3367) Delete topic dont delete the complete log from kafka
[ https://issues.apache.org/jira/browse/KAFKA-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189874#comment-15189874 ] Akshath Patkar commented on KAFKA-3367: --- [~mgharat] thanks. i am using confluent package 2.10.5. Issue we are running into is. We are running jdbc connector to replicate mysql data into kafka. In case if i have nuke data for a single table i should be delete the particualr topic(table) and run Bulk upload to load the complete data. But what happened is when i marked as deleted topic stayed there ansd says marled for deleted but it didnt get removed completely. When mysql connectors runs again it tries to use the same topic name. So is there any such thing is available where we use delete topic and it removed topic completely even name. Not just soft delete. > Delete topic dont delete the complete log from kafka > > > Key: KAFKA-3367 > URL: https://issues.apache.org/jira/browse/KAFKA-3367 > Project: Kafka > Issue Type: Improvement >Reporter: Akshath Patkar > > Delete topic Just marks the topic as deleted. But data still remain in logs. > How can we delete the topic completely with out doing manual delete of logs > from kafka and zookeeper -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3367) Delete topic dont delete the complete log from kafka
[ https://issues.apache.org/jira/browse/KAFKA-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190185#comment-15190185 ] Akshath Patkar commented on KAFKA-3367: --- Yes, Automatic topic creation is turned on. There is no consumers consuming the data. And yes kafka connectors which produces this data were stopped after the deletion of topic. > Delete topic dont delete the complete log from kafka > > > Key: KAFKA-3367 > URL: https://issues.apache.org/jira/browse/KAFKA-3367 > Project: Kafka > Issue Type: Improvement >Reporter: Akshath Patkar > > Delete topic Just marks the topic as deleted. But data still remain in logs. > How can we delete the topic completely with out doing manual delete of logs > from kafka and zookeeper -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3367) Delete topic dont delete the complete log from kafka
[ https://issues.apache.org/jira/browse/KAFKA-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193801#comment-15193801 ] Akshath Patkar commented on KAFKA-3367: --- Thanks [~mgharat] will close this issue for now. > Delete topic dont delete the complete log from kafka > > > Key: KAFKA-3367 > URL: https://issues.apache.org/jira/browse/KAFKA-3367 > Project: Kafka > Issue Type: Improvement >Reporter: Akshath Patkar > > Delete topic Just marks the topic as deleted. But data still remain in logs. > How can we delete the topic completely with out doing manual delete of logs > from kafka and zookeeper -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-4171) Kafka-connect prints outs keystone and truststore password in log2
Akshath Patkar created KAFKA-4171: - Summary: Kafka-connect prints outs keystone and truststore password in log2 Key: KAFKA-4171 URL: https://issues.apache.org/jira/browse/KAFKA-4171 Project: Kafka Issue Type: Improvement Components: KafkaConnect Affects Versions: 0.10.0.0 Reporter: Akshath Patkar Assignee: Ewen Cheslack-Postava Kafka-connect prints outs keystone and truststore password in log [2016-09-14 16:30:33,971] WARN The configuration consumer.ssl.truststore.password = X was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:186) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
