[jira] [Commented] (KAFKA-10457) JsonConverter.toConnectData trims BigInteger to Long for schema-less case
[
https://issues.apache.org/jira/browse/KAFKA-10457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438256#comment-17438256
]
Arjun Satish commented on KAFKA-10457:
--
looks like we would need a new [data
type|https://github.com/apache/kafka/blob/3.0.0/connect/api/src/main/java/org/apache/kafka/connect/data/Schema.java#L45-L107]
to accommodate large integers. Right now, the only precisions we support for
integers are: INT8, INT16, INT32, INT64.
Adding a new type will help handle it in
[JsonConverter|https://github.com/apache/kafka/blob/3.0.0/connect/json/src/main/java/org/apache/kafka/connect/json/JsonConverter.java#L70-L73]
correctly.
> JsonConverter.toConnectData trims BigInteger to Long for schema-less case
> -
>
> Key: KAFKA-10457
> URL: https://issues.apache.org/jira/browse/KAFKA-10457
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
>Priority: Critical
>
>
> When _JsonConverter_ is configured with _schemas.enable=false_ and value,
> exceeding _Double_ is passed, the result is incorrect since the converter
> trims it to _Double:_
> {code:java}
> Map props = Collections.singletonMap("schemas.enable",
> false);
> converter.configure(props, true);
> BigInteger value = BigInteger.valueOf(Long.MAX_VALUE).add(new
> BigInteger("1"));
> String msg = value.toString();
> SchemaAndValue schemaAndValue = converter.toConnectData(TOPIC,
> msg.getBytes());
> assertNull(schemaAndValue.schema());
> assertEquals(value, schemaAndValue.value());
> {code}
>
> Fails with:
>
> {code:java}
> expected:<9223372036854775808> but was:<-9223372036854775808>
> Expected :9223372036854775808
> Actual :-9223372036854775808
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-10387) Cannot include SMT configs with source connector that include topic.creation.* properties
[
https://issues.apache.org/jira/browse/KAFKA-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-10387:
-
Description:
Let's say we try to create a connector with the following config:
{code:java}
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "localhost",
"database.port": 32843,
"database.user": "test",
"database.password": "test",
"database.dbname": "test",
"database.server.name": "tcpsql",
"table.whitelist": "public.numerics",
"transforms": "abc",
"transforms.abc.type": "io.debezium.transforms.ExtractNewRecordState",
"topic.creation.default.partitions": "1",
"topic.creation.default.replication.factor": "1"
}
{code}
this fails with the following error in the Connector worker:
{code:java}
[2020-08-11 02:47:05,908] ERROR Failed to start task deb-0
(org.apache.kafka.connect.runtime.Worker:560)
org.apache.kafka.connect.errors.ConnectException:
org.apache.kafka.common.config.ConfigException: Unknown configuration
'transforms.abc.type'
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:296)
at
org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:605)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.apache.kafka.common.config.ConfigException: Unknown
configuration 'transforms.abc.type'
at
org.apache.kafka.common.config.AbstractConfig.get(AbstractConfig.java:159)
at
org.apache.kafka.connect.runtime.SourceConnectorConfig$EnrichedSourceConnectorConfig.get(SourceConnectorConfig.java:57)
at
org.apache.kafka.connect.runtime.SourceConnectorConfig.get(SourceConnectorConfig.java:141)
at
org.apache.kafka.common.config.AbstractConfig.getClass(AbstractConfig.java:216)
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:281)
... 10 more
{code}
connector creation works fine, if we remove the topic.creation properties
above.
Not entirely sure but it looks like the piece of code that might need a fix is
here (as it does not add {{transforms.*}} configs into the returned ConfigDef
instances:
https://github.com/apache/kafka/blob/2.6.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/SourceConnectorConfig.java#L94
was:
Let's say we try to create a connector with the following config:
{code:java}
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "localhost",
"database.port": 32843,
"database.user": "test",
"database.password": "test",
"database.dbname": "test",
"database.server.name": "tcpsql",
"table.whitelist": "public.numerics",
"transforms": "abc",
"transforms.abc.type": "io.debezium.transforms.ExtractNewRecordState",
"topic.creation.default.partitions": "1",
"topic.creation.default.replication.factor": "1"
}
{code}
this fails with the following error in the Connector worker:
{code:java}
[2020-08-11 02:47:05,908] ERROR Failed to start task deb-0
(org.apache.kafka.connect.runtime.Worker:560)
org.apache.kafka.connect.errors.ConnectException:
org.apache.kafka.common.config.ConfigException: Unknown configuration
'transforms.abc.type'
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:296)
at
org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:605)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
[jira] [Updated] (KAFKA-10387) Cannot include SMT configs with source connector that include topic.creation.* properties
[
https://issues.apache.org/jira/browse/KAFKA-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-10387:
-
Description:
Let's say we try to create a connector with the following config:
{code:java}
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "localhost",
"database.port": 32843,
"database.user": "test",
"database.password": "test",
"database.dbname": "test",
"database.server.name": "tcpsql",
"table.whitelist": "public.numerics",
"transforms": "abc",
"transforms.abc.type": "io.debezium.transforms.ExtractNewRecordState",
"topic.creation.default.partitions": "1",
"topic.creation.default.replication.factor": "1"
}
{code}
this fails with the following error in the Connector worker:
{code:java}
[2020-08-11 02:47:05,908] ERROR Failed to start task deb-0
(org.apache.kafka.connect.runtime.Worker:560)
org.apache.kafka.connect.errors.ConnectException:
org.apache.kafka.common.config.ConfigException: Unknown configuration
'transforms.abc.type'
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:296)
at
org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:605)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.apache.kafka.common.config.ConfigException: Unknown
configuration 'transforms.abc.type'
at
org.apache.kafka.common.config.AbstractConfig.get(AbstractConfig.java:159)
at
org.apache.kafka.connect.runtime.SourceConnectorConfig$EnrichedSourceConnectorConfig.get(SourceConnectorConfig.java:57)
at
org.apache.kafka.connect.runtime.SourceConnectorConfig.get(SourceConnectorConfig.java:141)
at
org.apache.kafka.common.config.AbstractConfig.getClass(AbstractConfig.java:216)
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:281)
... 10 more
{code}
connector creation works fine, if we remove the topic.creation properties
above.
Not entirely sure but it looks like the piece of code that might need a fix is
here (as it does not add transforms.* configs when creating
EnrichedConnectorConfig instances:
https://github.com/apache/kafka/blob/2.6.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/SourceConnectorConfig.java#L94
was:
Let's say we try to create a connector with the following config:
{{{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "localhost",
"database.port": 32843,
"database.user": "test",
"database.password": "test",
"database.dbname": "test",
"database.server.name": "tcpsql",
"table.whitelist": "public.numerics",
"transforms": "abc",
"transforms.abc.type": "io.debezium.transforms.ExtractNewRecordState",
"topic.creation.default.partitions": "1",
"topic.creation.default.replication.factor": "1"
}}}
this fails with the following error in the Connector worker:
{{
[2020-08-11 02:47:05,908] ERROR Failed to start task deb-0
(org.apache.kafka.connect.runtime.Worker:560)
org.apache.kafka.connect.errors.ConnectException:
org.apache.kafka.common.config.ConfigException: Unknown configuration
'transforms.abc.type'
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:296)
at
org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:605)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
[jira] [Updated] (KAFKA-10387) Cannot include SMT configs with source connector that include topic.creation.* properties
[
https://issues.apache.org/jira/browse/KAFKA-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-10387:
-
Description:
Let's say we try to create a connector with the following config:
{{{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "localhost",
"database.port": 32843,
"database.user": "test",
"database.password": "test",
"database.dbname": "test",
"database.server.name": "tcpsql",
"table.whitelist": "public.numerics",
"transforms": "abc",
"transforms.abc.type": "io.debezium.transforms.ExtractNewRecordState",
"topic.creation.default.partitions": "1",
"topic.creation.default.replication.factor": "1"
}}}
this fails with the following error in the Connector worker:
{{
[2020-08-11 02:47:05,908] ERROR Failed to start task deb-0
(org.apache.kafka.connect.runtime.Worker:560)
org.apache.kafka.connect.errors.ConnectException:
org.apache.kafka.common.config.ConfigException: Unknown configuration
'transforms.abc.type'
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:296)
at
org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:605)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.apache.kafka.common.config.ConfigException: Unknown
configuration 'transforms.abc.type'
at
org.apache.kafka.common.config.AbstractConfig.get(AbstractConfig.java:159)
at
org.apache.kafka.connect.runtime.SourceConnectorConfig$EnrichedSourceConnectorConfig.get(SourceConnectorConfig.java:57)
at
org.apache.kafka.connect.runtime.SourceConnectorConfig.get(SourceConnectorConfig.java:141)
at
org.apache.kafka.common.config.AbstractConfig.getClass(AbstractConfig.java:216)
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:281)
... 10 more
}}
connector creation works fine, if we remove the topic.creation properties
above.
Not entirely sure but it looks like the piece of code that might need a fix is
here (as it does not add transforms.* configs when creating
EnrichedConnectorConfig instances:
https://github.com/apache/kafka/blob/2.6.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/SourceConnectorConfig.java#L94
was:
Let's say we try to create a connector with the following config:
{{
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "localhost",
"database.port": 32843,
"database.user": "test",
"database.password": "test",
"database.dbname": "test",
"database.server.name": "tcpsql",
"table.whitelist": "public.numerics",
"transforms": "abc",
"transforms.abc.type": "io.debezium.transforms.ExtractNewRecordState",
"topic.creation.default.partitions": "1",
"topic.creation.default.replication.factor": "1"
}
}}
this fails with the following error in the Connector worker:
{{
[2020-08-11 02:47:05,908] ERROR Failed to start task deb-0
(org.apache.kafka.connect.runtime.Worker:560)
org.apache.kafka.connect.errors.ConnectException:
org.apache.kafka.common.config.ConfigException: Unknown configuration
'transforms.abc.type'
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:296)
at
org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:605)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
[jira] [Updated] (KAFKA-10387) Cannot include SMT configs with source connector that include topic.creation.* properties
[
https://issues.apache.org/jira/browse/KAFKA-10387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-10387:
-
Description:
Let's say we try to create a connector with the following config:
{{
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "localhost",
"database.port": 32843,
"database.user": "test",
"database.password": "test",
"database.dbname": "test",
"database.server.name": "tcpsql",
"table.whitelist": "public.numerics",
"transforms": "abc",
"transforms.abc.type": "io.debezium.transforms.ExtractNewRecordState",
"topic.creation.default.partitions": "1",
"topic.creation.default.replication.factor": "1"
}
}}
this fails with the following error in the Connector worker:
{{
[2020-08-11 02:47:05,908] ERROR Failed to start task deb-0
(org.apache.kafka.connect.runtime.Worker:560)
org.apache.kafka.connect.errors.ConnectException:
org.apache.kafka.common.config.ConfigException: Unknown configuration
'transforms.abc.type'
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:296)
at
org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:605)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.apache.kafka.common.config.ConfigException: Unknown
configuration 'transforms.abc.type'
at
org.apache.kafka.common.config.AbstractConfig.get(AbstractConfig.java:159)
at
org.apache.kafka.connect.runtime.SourceConnectorConfig$EnrichedSourceConnectorConfig.get(SourceConnectorConfig.java:57)
at
org.apache.kafka.connect.runtime.SourceConnectorConfig.get(SourceConnectorConfig.java:141)
at
org.apache.kafka.common.config.AbstractConfig.getClass(AbstractConfig.java:216)
at
org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:281)
... 10 more
}}
connector creation works fine, if we remove the topic.creation properties
above.
Not entirely sure but it looks like the piece of code that might need a fix is
here (as it does not add transforms.* configs when creating
EnrichedConnectorConfig instances:
https://github.com/apache/kafka/blob/2.6.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/SourceConnectorConfig.java#L94
> Cannot include SMT configs with source connector that include
> topic.creation.* properties
> -
>
> Key: KAFKA-10387
> URL: https://issues.apache.org/jira/browse/KAFKA-10387
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.6.0
>Reporter: Arjun Satish
>Priority: Major
>
> Let's say we try to create a connector with the following config:
> {{
> {
> "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
> "tasks.max": "1",
> "database.hostname": "localhost",
> "database.port": 32843,
> "database.user": "test",
> "database.password": "test",
> "database.dbname": "test",
> "database.server.name": "tcpsql",
> "table.whitelist": "public.numerics",
> "transforms": "abc",
> "transforms.abc.type": "io.debezium.transforms.ExtractNewRecordState",
> "topic.creation.default.partitions": "1",
> "topic.creation.default.replication.factor": "1"
> }
> }}
> this fails with the following error in the Connector worker:
> {{
> [2020-08-11 02:47:05,908] ERROR Failed to start task deb-0
> (org.apache.kafka.connect.runtime.Worker:560)
> org.apache.kafka.connect.errors.ConnectException:
> org.apache.kafka.common.config.ConfigException: Unknown configuration
> 'transforms.abc.type'
> at
> org.apache.kafka.connect.runtime.ConnectorConfig.transformations(ConnectorConfig.java:296)
> at
> org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:605)
> at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)
> at
>
[jira] [Created] (KAFKA-10387) Cannot include SMT configs with source connector that include topic.creation.* properties
Arjun Satish created KAFKA-10387: Summary: Cannot include SMT configs with source connector that include topic.creation.* properties Key: KAFKA-10387 URL: https://issues.apache.org/jira/browse/KAFKA-10387 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 2.6.0 Reporter: Arjun Satish -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10340) Source connectors should report error when trying to producer records to non-existent topics instead of hanging forever
[
https://issues.apache.org/jira/browse/KAFKA-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-10340:
-
Description:
Currently, a source connector will blindly attempt to write a record to a Kafka
topic. When the topic does not exist, its creation is controlled by the
{{auto.create.topics.enable}} config on the brokers. When auto.create is
disabled, the producer.send() call on the Connect worker will hang indefinitely
(due to the "infinite retries" configuration for said producer). In setups
where this config is usually disabled, the source connector simply appears to
hang and not produce any output.
It is desirable to either log an info or an error message (or inform the user
somehow) that the connector is simply stuck waiting for the destination topic
to be created. When the worker has permissions to inspect the broker settings,
it can use the {{listTopics}} and {{describeConfigs}} API in AdminClient to
check if the topic exists, the broker can {{auto.create.topics.enable}} topics,
and if these cases do not exist, either throw an error.
With the recently merged
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
this becomes even more specific a corner case: when topic creation settings
are enabled, the worker should handle the corner case where topic creation is
disabled, {{auto.create.topics.enable}} is set to false and topic does not
exist.
was:
Currently, a source connector will blindly attempt to write a record to a Kafka
topic. When the topic does not exist, its creation is controlled by the
{{auto.create.topics.enable}} config on the brokers. When auto.create is
disabled, the producer.send() call on the Connect worker will hang indefinitely
(due to the "infinite retries" configuration for said producer). In production
setups, where this config is usually disabled, the source connector simply
appears to hang and not produce any output.
It is desirable to either log an info or an error message (or inform the user
somehow) that the connector is simply stuck waiting for the destination topic
to be created. When the worker has permissions to inspect the broker settings,
it can use the {{listTopics}} and {{describeConfigs}} API in AdminClient to
check if the topic exists, the broker can {{auto.create.topics.enable}} topics,
and if these cases do not exist, either throw an error.
With the recently merged
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
this becomes even more specific a corner case: when topic creation settings
are enabled, the worker should handle the corner case where topic creation is
disabled, {{auto.create.topics.enable}} is set to false and topic does not
exist.
> Source connectors should report error when trying to producer records to
> non-existent topics instead of hanging forever
> ---
>
> Key: KAFKA-10340
> URL: https://issues.apache.org/jira/browse/KAFKA-10340
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Arjun Satish
>Priority: Major
>
> Currently, a source connector will blindly attempt to write a record to a
> Kafka topic. When the topic does not exist, its creation is controlled by the
> {{auto.create.topics.enable}} config on the brokers. When auto.create is
> disabled, the producer.send() call on the Connect worker will hang
> indefinitely (due to the "infinite retries" configuration for said producer).
> In setups where this config is usually disabled, the source connector simply
> appears to hang and not produce any output.
> It is desirable to either log an info or an error message (or inform the user
> somehow) that the connector is simply stuck waiting for the destination topic
> to be created. When the worker has permissions to inspect the broker
> settings, it can use the {{listTopics}} and {{describeConfigs}} API in
> AdminClient to check if the topic exists, the broker can
> {{auto.create.topics.enable}} topics, and if these cases do not exist, either
> throw an error.
> With the recently merged
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
> this becomes even more specific a corner case: when topic creation settings
> are enabled, the worker should handle the corner case where topic creation is
> disabled, {{auto.create.topics.enable}} is set to false and topic does not
> exist.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-10340) Source connectors should report error when trying to producer records to non-existent topics instead of hanging forever
[
https://issues.apache.org/jira/browse/KAFKA-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-10340:
-
Description:
Currently, a source connector will blindly attempt to write a record to a Kafka
topic. When the topic does not exist, its creation is controlled by the
{{auto.create.topics.enable}} config on the brokers. When auto.create is
disabled, the producer.send() call on the Connect worker will hang indefinitely
(due to the "infinite retries" configuration for said producer). In production
setups, where this config is usually disabled, the source connector simply
appears to hang and not produce any output.
It is desirable to either log an info or an error message (or inform the user
somehow) that the connector is simply stuck waiting for the destination topic
to be created. When the worker has permissions to inspect the broker settings,
it can use the {{listTopics}} and {{describeConfigs}} API in AdminClient to
check if the topic exists, the broker can {{auto.create.topics.enable}} topics,
and if these cases do not exist, either throw an error.
With the recently merged
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
this becomes even more specific a corner case: when topic creation settings
are enabled, the worker should handle the corner case where topic creation is
disabled, {{auto.create.topics.enable}} is set to false and topic does not
exist.
was:
Currently, a source connector will blindly attempt to write a record to a Kafka
topic. When the topic does not exist, its creation is controlled by ACLs and
the {{auto.create.topics.enable}} config on the brokers. When auto.create is
disabled, the producer.send() call on the Connect worker will hang indefinitely
(due to the "infinite retries" configuration for said producer). In production
setups, the config is usually disabled, and the source connector simply appears
to hang and not produce any output.
It is desirable to either log an info or an error message (or inform the user
somehow) that the connector is simply stuck waiting for the destination topic
to be created. When the worker has permissions to inspect the broker settings,
it can use the {{listTopics}} and {{describeConfigs}} API in AdminClient to
check if the topic exists, the broker can {{auto.create.topics.enable}} topics,
and if these cases do not exist, either throw an error.
With the recently merged
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
this becomes even more specific a corner case: when topic creation settings
are enabled, the worker should handle the corner case where topic creation is
disabled, {{auto.create.topics.enable}} is set to false and topic does not
exist.
> Source connectors should report error when trying to producer records to
> non-existent topics instead of hanging forever
> ---
>
> Key: KAFKA-10340
> URL: https://issues.apache.org/jira/browse/KAFKA-10340
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Arjun Satish
>Priority: Major
>
> Currently, a source connector will blindly attempt to write a record to a
> Kafka topic. When the topic does not exist, its creation is controlled by the
> {{auto.create.topics.enable}} config on the brokers. When auto.create is
> disabled, the producer.send() call on the Connect worker will hang
> indefinitely (due to the "infinite retries" configuration for said producer).
> In production setups, where this config is usually disabled, the source
> connector simply appears to hang and not produce any output.
> It is desirable to either log an info or an error message (or inform the user
> somehow) that the connector is simply stuck waiting for the destination topic
> to be created. When the worker has permissions to inspect the broker
> settings, it can use the {{listTopics}} and {{describeConfigs}} API in
> AdminClient to check if the topic exists, the broker can
> {{auto.create.topics.enable}} topics, and if these cases do not exist, either
> throw an error.
> With the recently merged
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
> this becomes even more specific a corner case: when topic creation settings
> are enabled, the worker should handle the corner case where topic creation is
> disabled, {{auto.create.topics.enable}} is set to false and topic does not
> exist.
--
This message was sent by
[jira] [Created] (KAFKA-10340) Source connectors should report error when trying to producer records to non-existent topics instead of hanging forever
Arjun Satish created KAFKA-10340:
Summary: Source connectors should report error when trying to
producer records to non-existent topics instead of hanging forever
Key: KAFKA-10340
URL: https://issues.apache.org/jira/browse/KAFKA-10340
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Arjun Satish
Currently, a source connector will blindly attempt to write a record to a Kafka
topic. When the topic does not exist, its creation is controlled by ACLs and
the {{auto.create.topics.enable}} config on the brokers. When auto.create is
disabled, the producer.send() call on the Connect worker will hang indefinitely
(due to the "infinite retries" configuration for said producer). In production
setups, the config is usually disabled, and the source connector simply appears
to hang and not produce any output.
It is desirable to either log an info or an error message (or inform the user
somehow) that the connector is simply stuck waiting for the destination topic
to be created. When the worker has permissions to inspect the broker settings,
it can use the {{listTopics}} and {{describeConfigs}} API in AdminClient to
check if the topic exists, the broker can {{auto.create.topics.enable}} topics,
and if these cases do not exist, either throw an error.
With the recently merged
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
this becomes even more specific a corner case: when topic creation settings
are enabled, the worker should handle the corner case where topic creation is
disabled, {{auto.create.topics.enable}} is set to false and topic does not
exist.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10021) When reading to the end of the config log, check if fetch.max.wait.ms is greater than worker.sync.timeout.ms
[
https://issues.apache.org/jira/browse/KAFKA-10021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111584#comment-17111584
]
Arjun Satish commented on KAFKA-10021:
--
Thanks, [~hachikuji]!
> When reading to the end of the config log, check if fetch.max.wait.ms is
> greater than worker.sync.timeout.ms
>
>
> Key: KAFKA-10021
> URL: https://issues.apache.org/jira/browse/KAFKA-10021
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Sanjana Kaundinya
>Priority: Major
>
> Currently in the Connect code in DistributedHerder.java, we see the following
> piece of code
>
> {{if (!canReadConfigs && !readConfigToEnd(workerSyncTimeoutMs))
> return; // Safe to return and tick immediately because
> readConfigToEnd will do the backoff for us}}
> where the workerSyncTimeoutMs passed in is the timeout given to read to the
> end of the config log. This is a bug as we should check if fetch.wait.max.ms
> is greater than worker.sync.timeout.ms and if it is, use
> worker.sync.timeout.ms as the fetch.wait.max.ms. A better fix would be to use
> the AdminClient to read to the end of the log, but at a minimum we should
> check the configs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-10021) When reading to the end of the config log, check if fetch.max.wait.ms is greater than worker.sync.timeout.ms
[
https://issues.apache.org/jira/browse/KAFKA-10021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111493#comment-17111493
]
Arjun Satish commented on KAFKA-10021:
--
[~hachikuji] [~skaundinya] do consumers in a JVM share a connection to a broker
or do they all create their own connections? if they share a connection, then
this problem can occur if a connector/task has its consumer overrides that sets
a high `fetch.max.wait.ms`. In this case, the worker should not allow
overriding this value in a connector to more than what is allowed by the
workerSyncTimeoutMs.
> When reading to the end of the config log, check if fetch.max.wait.ms is
> greater than worker.sync.timeout.ms
>
>
> Key: KAFKA-10021
> URL: https://issues.apache.org/jira/browse/KAFKA-10021
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Sanjana Kaundinya
>Priority: Major
>
> Currently in the Connect code in DistributedHerder.java, we see the following
> piece of code
>
> {{if (!canReadConfigs && !readConfigToEnd(workerSyncTimeoutMs))
> return; // Safe to return and tick immediately because
> readConfigToEnd will do the backoff for us}}
> where the workerSyncTimeoutMs passed in is the timeout given to read to the
> end of the config log. This is a bug as we should check if fetch.wait.max.ms
> is greater than worker.sync.timeout.ms and if it is, use
> worker.sync.timeout.ms as the fetch.wait.max.ms. A better fix would be to use
> the AdminClient to read to the end of the log, but at a minimum we should
> check the configs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8774) Connect REST API exposes plaintext secrets in tasks endpoint if config value contains additional characters
[
https://issues.apache.org/jira/browse/KAFKA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-8774:
Summary: Connect REST API exposes plaintext secrets in tasks endpoint if
config value contains additional characters (was: Connect REST API exposes
plaintext secrets in tasks endpoint)
> Connect REST API exposes plaintext secrets in tasks endpoint if config value
> contains additional characters
> ---
>
> Key: KAFKA-8774
> URL: https://issues.apache.org/jira/browse/KAFKA-8774
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0
>Reporter: Oleksandr Diachenko
>Assignee: Oleksandr Diachenko
>Priority: Critical
>
> I have configured a Connector to use externalized secrets, and the following
> endpoint returns secrets in the externalized form:
> {code:java}
> curl localhost:8083/connectors/foobar|jq
> {code}
> {code:java}
> {
> "name": "foobar",
> "config": {
> "connector.class": "io.confluent.connect.s3.S3SinkConnector",
> ...
> "consumer.override.sasl.jaas.config":
> "org.apache.kafka.common.security.plain.PlainLoginModule required
> username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
> password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
> "admin.override.sasl.jaas.config":
> "org.apache.kafka.common.security.plain.PlainLoginModule required
> username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
> password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
> "consumer.sasl.jaas.config":
> "org.apache.kafka.common.security.plain.PlainLoginModule required
> username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
> password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
> "producer.override.sasl.jaas.config":
> "org.apache.kafka.common.security.plain.PlainLoginModule required
> username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
> password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
> "producer.sasl.jaas.config":
> "org.apache.kafka.common.security.plain.PlainLoginModule required
> username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
> password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
> ...
> },
> "tasks": [
> { "connector": "foobar", "task": 0 }
> ],
> "type": "sink"
> }{code}
> But another endpoint returns secrets in plain text:
> {code:java}
> curl localhost:8083/connectors/foobar/tasks|jq
> {code}
> {code:java}
> [
> {
> "id": {
> "connector": "lcc-kgkpm",
> "task": 0
> },
> "config": {
> "connector.class": "io.confluent.connect.s3.S3SinkConnector",
> ...
> "errors.log.include.messages": "true",
> "flush.size": "1000",
> "consumer.override.sasl.jaas.config":
> "org.apache.kafka.common.security.plain.PlainLoginModule required
> username=\"OOPS\" password=\"SURPRISE\";",
> "admin.override.sasl.jaas.config":
> "org.apache.kafka.common.security.plain.PlainLoginModule required
> username=\"OOPS\" password=\"SURPRISE\";",
> "consumer.sasl.jaas.config":
> "org.apache.kafka.common.security.plain.PlainLoginModule required
> username=\"OOPS\" password=\"SURPRISE\";",
> "producer.override.sasl.jaas.config":
> "org.apache.kafka.common.security.plain.PlainLoginModule required
> username=\"OOPS\" password=\"SURPRISE\";",
> "producer.sasl.jaas.config":
> "org.apache.kafka.common.security.plain.PlainLoginModule required
> username=\"OOPS\" password=\"SURPRISE\";",
> ...
> }
> }
> ]
> {code}
>
> EDIT: This bug only shows up if the secrets are a substring in the config
> value. If they form the entirety of the config value, then the secrets are
> hidden at the /tasks endpoints.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (KAFKA-8774) Connect REST API exposes plaintext secrets in tasks endpoint
[
https://issues.apache.org/jira/browse/KAFKA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-8774:
Description:
I have configured a Connector to use externalized secrets, and the following
endpoint returns secrets in the externalized form:
{code:java}
curl localhost:8083/connectors/foobar|jq
{code}
{code:java}
{
"name": "foobar",
"config": {
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
...
"consumer.override.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
"admin.override.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
"consumer.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
"producer.override.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
"producer.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
...
},
"tasks": [
{ "connector": "foobar", "task": 0 }
],
"type": "sink"
}{code}
But another endpoint returns secrets in plain text:
{code:java}
curl localhost:8083/connectors/foobar/tasks|jq
{code}
{code:java}
[
{
"id": {
"connector": "lcc-kgkpm",
"task": 0
},
"config": {
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
...
"errors.log.include.messages": "true",
"flush.size": "1000",
"consumer.override.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"OOPS\" password=\"SURPRISE\";",
"admin.override.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"OOPS\" password=\"SURPRISE\";",
"consumer.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"OOPS\" password=\"SURPRISE\";",
"producer.override.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"OOPS\" password=\"SURPRISE\";",
"producer.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"OOPS\" password=\"SURPRISE\";",
...
}
}
]
{code}
EDIT: This bug only shows up if the secrets are a substring in the config
value. If they form the entirety of the config value, then the secrets are
hidden at the /tasks endpoints.
was:
I have configured a Connector to use externalized secrets, and the following
endpoint returns secrets in the externalized form:
{code:java}
curl localhost:8083/connectors/foobar|jq
{code}
{code:java}
{
"name": "foobar",
"config": {
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
...
"consumer.override.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
"admin.override.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
"consumer.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
"producer.override.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
"producer.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\"
password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
...
},
"tasks": [
{ "connector": "foobar", "task": 0 }
],
"type": "sink"
}{code}
But another endpoint returns secrets in plain text:
{code:java}
curl localhost:8083/connectors/foobar/tasks|jq
{code}
{code:java}
[
{
[jira] [Commented] (KAFKA-8597) Give access to the Dead Letter Queue APIs to Kafka Connect Developers
[ https://issues.apache.org/jira/browse/KAFKA-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876309#comment-16876309 ] Arjun Satish commented on KAFKA-8597: - Thanks for the feature request, [~conker84]. I think there might be more value in integrating the reporter framework [1], so we can tap into existing reporters (currently, we have two, including the DLQ). That way, if a DLQ is disabled by the end-user, calls to report(...) will respect that. But as Randall mentioned, introducing additional APIs comes with a catch. An alternative could be to raise a new Exception class (say, BadRecordException that extends RetriableException) that can inform the framework of bad records. Here, we might only have to think about how to serialize records into the DLQ (the framework might not be holding on to these original records if the connector buffers records). [1] [https://github.com/apache/kafka/blob/2.3.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/errors/RetryWithToleranceOperator.java#L245-L247 ] > Give access to the Dead Letter Queue APIs to Kafka Connect Developers > - > > Key: KAFKA-8597 > URL: https://issues.apache.org/jira/browse/KAFKA-8597 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Andrea Santurbano >Priority: Major > Fix For: 2.4.0 > > > Would be cool to have the chance to have access to the DLQ APIs in order to > enable us (developers) to use that. > For instance, if someone uses JSON as message format with no schema and it's > trying to import some data into a table, and the JSON contains a null value > for a NON-NULL table field, so we want to move that event to the DLQ. > Thanks a lot! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-8597) Give access to the Dead Letter Queue APIs to Kafka Connect Developers
[ https://issues.apache.org/jira/browse/KAFKA-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876309#comment-16876309 ] Arjun Satish edited comment on KAFKA-8597 at 7/1/19 4:10 PM: - Thanks for the feature request, [~conker84]. I think there might be more value in integrating the reporter framework [1], so we can tap into existing reporters (currently, we have two, including the DLQ). That way, if a DLQ is disabled by the end-user, calls to report(...) will respect that. But as Randall mentioned, introducing additional APIs comes with a catch. An alternative could be to raise a new Exception class (say, BadRecordException that extends RetriableException) that can inform the framework of bad records. Here, we might only have to think about how to serialize records into the DLQ (the framework might not be holding on to these original records if the connector buffers records). [1] [https://github.com/apache/kafka/blob/2.3.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/errors/RetryWithToleranceOperator.java#L245-L247 was (Author: wicknicks): Thanks for the feature request, [~conker84]. I think there might be more value in integrating the reporter framework [1], so we can tap into existing reporters (currently, we have two, including the DLQ). That way, if a DLQ is disabled by the end-user, calls to report(...) will respect that. But as Randall mentioned, introducing additional APIs comes with a catch. An alternative could be to raise a new Exception class (say, BadRecordException that extends RetriableException) that can inform the framework of bad records. Here, we might only have to think about how to serialize records into the DLQ (the framework might not be holding on to these original records if the connector buffers records). [1] [https://github.com/apache/kafka/blob/2.3.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/errors/RetryWithToleranceOperator.java#L245-L247 ] > Give access to the Dead Letter Queue APIs to Kafka Connect Developers > - > > Key: KAFKA-8597 > URL: https://issues.apache.org/jira/browse/KAFKA-8597 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Andrea Santurbano >Priority: Major > Fix For: 2.4.0 > > > Would be cool to have the chance to have access to the DLQ APIs in order to > enable us (developers) to use that. > For instance, if someone uses JSON as message format with no schema and it's > trying to import some data into a table, and the JSON contains a null value > for a NON-NULL table field, so we want to move that event to the DLQ. > Thanks a lot! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7877) Connect DLQ not used in SinkTask put()
[ https://issues.apache.org/jira/browse/KAFKA-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish resolved KAFKA-7877. - Resolution: Fixed Updated KIP. > Connect DLQ not used in SinkTask put() > -- > > Key: KAFKA-7877 > URL: https://issues.apache.org/jira/browse/KAFKA-7877 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.0.0, 2.0.1, 2.1.0 >Reporter: Andrew Bourgeois >Assignee: Arjun Satish >Priority: Major > > The Dead Letter Queue isn't implemented for the put() operation. > In > [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java] > the "retryWithToleranceOperator" gets used during the conversion and > transformation phases, but not when delivering the messages to the sink task. > According to KIP-298, the Dead Letter Queue should be used as long as we > throw org.apache.kafka.connect.errors.RetriableException. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-7877) Connect DLQ not used in SinkTask put()
[ https://issues.apache.org/jira/browse/KAFKA-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16828808#comment-16828808 ] Arjun Satish edited comment on KAFKA-7877 at 4/28/19 6:23 PM: -- Resolving this ticket. Feel free to reopen if we need some more clarifying text. was (Author: wicknicks): Updated KIP. > Connect DLQ not used in SinkTask put() > -- > > Key: KAFKA-7877 > URL: https://issues.apache.org/jira/browse/KAFKA-7877 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.0.0, 2.0.1, 2.1.0 >Reporter: Andrew Bourgeois >Assignee: Arjun Satish >Priority: Major > > The Dead Letter Queue isn't implemented for the put() operation. > In > [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java] > the "retryWithToleranceOperator" gets used during the conversion and > transformation phases, but not when delivering the messages to the sink task. > According to KIP-298, the Dead Letter Queue should be used as long as we > throw org.apache.kafka.connect.errors.RetriableException. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7877) Connect DLQ not used in SinkTask put()
[ https://issues.apache.org/jira/browse/KAFKA-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish reassigned KAFKA-7877: --- Assignee: Arjun Satish > Connect DLQ not used in SinkTask put() > -- > > Key: KAFKA-7877 > URL: https://issues.apache.org/jira/browse/KAFKA-7877 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.0.0, 2.0.1, 2.1.0 >Reporter: Andrew Bourgeois >Assignee: Arjun Satish >Priority: Major > > The Dead Letter Queue isn't implemented for the put() operation. > In > [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java] > the "retryWithToleranceOperator" gets used during the conversion and > transformation phases, but not when delivering the messages to the sink task. > According to KIP-298, the Dead Letter Queue should be used as long as we > throw org.apache.kafka.connect.errors.RetriableException. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7999) Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
[
https://issues.apache.org/jira/browse/KAFKA-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish resolved KAFKA-7999.
-
Resolution: Fixed
PR: https://github.com/apache/kafka/pull/6326
> Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
>
>
> Key: KAFKA-7999
> URL: https://issues.apache.org/jira/browse/KAFKA-7999
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Arjun Satish
>Priority: Critical
> Fix For: 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/30/]
> {quote}org.apache.kafka.common.KafkaException: Could not produce message to
> topic=test-topic at
> org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.produce(EmbeddedKafkaCluster.java:257)
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testProduceConsumeConnector(ExampleConnectIntegrationTest.java:129){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7999) Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
[
https://issues.apache.org/jira/browse/KAFKA-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778230#comment-16778230
]
Arjun Satish commented on KAFKA-7999:
-
https://github.com/apache/kafka/pull/6326 was merged to address this issue. We
were not able to reproduce it locally, and suspect that this failure happens
when many tests run in parallel on low resource VMs. Increasing the timeout (as
in the linked PR) should reduce the chances of random failure.
> Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
>
>
> Key: KAFKA-7999
> URL: https://issues.apache.org/jira/browse/KAFKA-7999
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Arjun Satish
>Priority: Critical
> Fix For: 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/30/]
> {quote}org.apache.kafka.common.KafkaException: Could not produce message to
> topic=test-topic at
> org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.produce(EmbeddedKafkaCluster.java:257)
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testProduceConsumeConnector(ExampleConnectIntegrationTest.java:129){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7999) Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
[
https://issues.apache.org/jira/browse/KAFKA-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish reassigned KAFKA-7999:
---
Assignee: Arjun Satish
> Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
>
>
> Key: KAFKA-7999
> URL: https://issues.apache.org/jira/browse/KAFKA-7999
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Arjun Satish
>Priority: Critical
> Fix For: 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/30/]
> {quote}org.apache.kafka.common.KafkaException: Could not produce message to
> topic=test-topic at
> org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.produce(EmbeddedKafkaCluster.java:257)
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testProduceConsumeConnector(ExampleConnectIntegrationTest.java:129){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7909) Ensure timely rebalance completion after pending members rejoin or fail
[ https://issues.apache.org/jira/browse/KAFKA-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-7909: Summary: Ensure timely rebalance completion after pending members rejoin or fail (was: Coordinator changes cause Connect integration test to fail) > Ensure timely rebalance completion after pending members rejoin or fail > --- > > Key: KAFKA-7909 > URL: https://issues.apache.org/jira/browse/KAFKA-7909 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 2.2.0 >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Blocker > Fix For: 2.2.0 > > > We recently introduced integration tests in Connect. This test spins up one > or more Connect workers along with a Kafka broker and Zk in a single process > and attempts to move records using a Connector. In the [Example Integration > Test|https://github.com/apache/kafka/blob/3c73633/connect/runtime/src/test/java/org/apache/kafka/connect/integration/ExampleConnectIntegrationTest.java#L105], > we spin up three workers each hosting a Connector task that consumes records > from a Kafka topic. When the connector starts up, it may go through multiple > rounds of rebalancing. We notice the following two problems in the last few > days: > # After members join a group, there are no pendingMembers remaining, but the > join group method does not complete, and send these members a signal that > they are not ready to start consuming from their respective partitions. > # Because of quick rebalances, a consumer might have started a group, but > Connect starts a rebalance, after we which we create three new instances of > the consumer (one from each worker/task). But the group coordinator seems to > have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Even though this ticket is described in the connect of Connect, it may be > applicable to general consumers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7909) Coordinator changes cause Connect integration test to fail
[ https://issues.apache.org/jira/browse/KAFKA-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish reassigned KAFKA-7909: --- Assignee: Arjun Satish > Coordinator changes cause Connect integration test to fail > -- > > Key: KAFKA-7909 > URL: https://issues.apache.org/jira/browse/KAFKA-7909 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 2.2.0 >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Blocker > Fix For: 2.2.0 > > > We recently introduced integration tests in Connect. This test spins up one > or more Connect workers along with a Kafka broker and Zk in a single process > and attempts to move records using a Connector. In the [Example Integration > Test|https://github.com/apache/kafka/blob/3c73633/connect/runtime/src/test/java/org/apache/kafka/connect/integration/ExampleConnectIntegrationTest.java#L105], > we spin up three workers each hosting a Connector task that consumes records > from a Kafka topic. When the connector starts up, it may go through multiple > rounds of rebalancing. We notice the following two problems in the last few > days: > # After members join a group, there are no pendingMembers remaining, but the > join group method does not complete, and send these members a signal that > they are not ready to start consuming from their respective partitions. > # Because of quick rebalances, a consumer might have started a group, but > Connect starts a rebalance, after we which we create three new instances of > the consumer (one from each worker/task). But the group coordinator seems to > have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Even though this ticket is described in the connect of Connect, it may be > applicable to general consumers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7909) Coordinator changes cause Connect integration test to fail
[ https://issues.apache.org/jira/browse/KAFKA-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763812#comment-16763812 ] Arjun Satish commented on KAFKA-7909: - [~ijuma] I believe [this commit|https://github.com/apache/kafka/commit/9a9310d074ead70ebf3e93d29d880e094b9080f6] caused the regression. > Coordinator changes cause Connect integration test to fail > -- > > Key: KAFKA-7909 > URL: https://issues.apache.org/jira/browse/KAFKA-7909 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 2.2.0 >Reporter: Arjun Satish >Priority: Blocker > Fix For: 2.2.0 > > > We recently introduced integration tests in Connect. This test spins up one > or more Connect workers along with a Kafka broker and Zk in a single process > and attempts to move records using a Connector. In the [Example Integration > Test|https://github.com/apache/kafka/blob/3c73633/connect/runtime/src/test/java/org/apache/kafka/connect/integration/ExampleConnectIntegrationTest.java#L105], > we spin up three workers each hosting a Connector task that consumes records > from a Kafka topic. When the connector starts up, it may go through multiple > rounds of rebalancing. We notice the following two problems in the last few > days: > # After members join a group, there are no pendingMembers remaining, but the > join group method does not complete, and send these members a signal that > they are not ready to start consuming from their respective partitions. > # Because of quick rebalances, a consumer might have started a group, but > Connect starts a rebalance, after we which we create three new instances of > the consumer (one from each worker/task). But the group coordinator seems to > have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Even though this ticket is described in the connect of Connect, it may be > applicable to general consumers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7909) Coordinator changes cause Connect integration test to fail
[ https://issues.apache.org/jira/browse/KAFKA-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-7909: Component/s: core > Coordinator changes cause Connect integration test to fail > -- > > Key: KAFKA-7909 > URL: https://issues.apache.org/jira/browse/KAFKA-7909 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.2.0 >Reporter: Arjun Satish >Priority: Blocker > Fix For: 2.2.0 > > > We recently introduced integration tests in Connect. This test spins up one > or more Connect workers along with a Kafka broker and Zk in a single process > and attempts to move records using a Connector. In the Example Integration > Test, we spin up three workers each hosting a Connector task that consumes > records from a Kafka topic. When the connector starts up, it may go through > multiple rounds of rebalancing. We notice the following two problems in the > last few days: > # After members join a group, there are no pendingMembers remaining, but the > join group method does not complete, and send these members a signal that > they are not ready to start consuming from their respective partitions. > # Because of quick rebalances, a consumer might have started a group, but > Connect starts a rebalance, after we which we create three new instances of > the consumer (one from each worker/task). But the group coordinator seems to > have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Even though this ticket is described in the connect of Connect, it may be > applicable to general consumers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7909) Coordinator changes cause Connect integration test to fail
[ https://issues.apache.org/jira/browse/KAFKA-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-7909: Description: We recently introduced integration tests in Connect. This test spins up one or more Connect workers along with a Kafka broker and Zk in a single process and attempts to move records using a Connector. In the [Example Integration Test|https://github.com/apache/kafka/blob/3c73633/connect/runtime/src/test/java/org/apache/kafka/connect/integration/ExampleConnectIntegrationTest.java#L105], we spin up three workers each hosting a Connector task that consumes records from a Kafka topic. When the connector starts up, it may go through multiple rounds of rebalancing. We notice the following two problems in the last few days: # After members join a group, there are no pendingMembers remaining, but the join group method does not complete, and send these members a signal that they are not ready to start consuming from their respective partitions. # Because of quick rebalances, a consumer might have started a group, but Connect starts a rebalance, after we which we create three new instances of the consumer (one from each worker/task). But the group coordinator seems to have 4 members in the group. This causes the JoinGroup to indefinitely stall. Even though this ticket is described in the connect of Connect, it may be applicable to general consumers. was: We recently introduced integration tests in Connect. This test spins up one or more Connect workers along with a Kafka broker and Zk in a single process and attempts to move records using a Connector. In the Example Integration Test, we spin up three workers each hosting a Connector task that consumes records from a Kafka topic. When the connector starts up, it may go through multiple rounds of rebalancing. We notice the following two problems in the last few days: # After members join a group, there are no pendingMembers remaining, but the join group method does not complete, and send these members a signal that they are not ready to start consuming from their respective partitions. # Because of quick rebalances, a consumer might have started a group, but Connect starts a rebalance, after we which we create three new instances of the consumer (one from each worker/task). But the group coordinator seems to have 4 members in the group. This causes the JoinGroup to indefinitely stall. Even though this ticket is described in the connect of Connect, it may be applicable to general consumers. > Coordinator changes cause Connect integration test to fail > -- > > Key: KAFKA-7909 > URL: https://issues.apache.org/jira/browse/KAFKA-7909 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 2.2.0 >Reporter: Arjun Satish >Priority: Blocker > Fix For: 2.2.0 > > > We recently introduced integration tests in Connect. This test spins up one > or more Connect workers along with a Kafka broker and Zk in a single process > and attempts to move records using a Connector. In the [Example Integration > Test|https://github.com/apache/kafka/blob/3c73633/connect/runtime/src/test/java/org/apache/kafka/connect/integration/ExampleConnectIntegrationTest.java#L105], > we spin up three workers each hosting a Connector task that consumes records > from a Kafka topic. When the connector starts up, it may go through multiple > rounds of rebalancing. We notice the following two problems in the last few > days: > # After members join a group, there are no pendingMembers remaining, but the > join group method does not complete, and send these members a signal that > they are not ready to start consuming from their respective partitions. > # Because of quick rebalances, a consumer might have started a group, but > Connect starts a rebalance, after we which we create three new instances of > the consumer (one from each worker/task). But the group coordinator seems to > have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Even though this ticket is described in the connect of Connect, it may be > applicable to general consumers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7909) Coordinator changes cause Connect integration test to fail
[ https://issues.apache.org/jira/browse/KAFKA-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-7909: Component/s: consumer > Coordinator changes cause Connect integration test to fail > -- > > Key: KAFKA-7909 > URL: https://issues.apache.org/jira/browse/KAFKA-7909 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 2.2.0 >Reporter: Arjun Satish >Priority: Blocker > Fix For: 2.2.0 > > > We recently introduced integration tests in Connect. This test spins up one > or more Connect workers along with a Kafka broker and Zk in a single process > and attempts to move records using a Connector. In the Example Integration > Test, we spin up three workers each hosting a Connector task that consumes > records from a Kafka topic. When the connector starts up, it may go through > multiple rounds of rebalancing. We notice the following two problems in the > last few days: > # After members join a group, there are no pendingMembers remaining, but the > join group method does not complete, and send these members a signal that > they are not ready to start consuming from their respective partitions. > # Because of quick rebalances, a consumer might have started a group, but > Connect starts a rebalance, after we which we create three new instances of > the consumer (one from each worker/task). But the group coordinator seems to > have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Even though this ticket is described in the connect of Connect, it may be > applicable to general consumers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7909) Coordinator changes cause Connect integration test to fail
[ https://issues.apache.org/jira/browse/KAFKA-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-7909: Description: We recently introduced integration tests in Connect. This test spins up one or more Connect workers along with a Kafka broker and Zk in a single process and attempts to move records using a Connector. In the Example Integration Test, we spin up three workers each hosting a Connector task that consumes records from a Kafka topic. When the connector starts up, it may go through multiple rounds of rebalancing. We notice the following two problems in the last few days: # After members join a group, there are no pendingMembers remaining, but the join group method does not complete, and send these members a signal that they are not ready to start consuming from their respective partitions. # Because of quick rebalances, a consumer might have started a group, but Connect starts a rebalance, after we which we create three new instances of the consumer (one from each worker/task). But the group coordinator seems to have 4 members in the group. This causes the JoinGroup to indefinitely stall. Even though this ticket is described in the connect of Connect, it may be applicable to general consumers. was: We recently introduced integration tests in Connect. This test spins up one or more Connect workers along with a Kafka broker and Zk in a single process and attempts to move records using a Connector. In the Example Integration Test, we spin up three workers each hosting a Connector task that consumes records from a Kafka topic. When the connector starts up, it may go through multiple rounds of rebalancing. We notice the following two problems in the last few days: # After members join a group, there are no pendingMembers remaining, but the join group method does not complete, and send these members a signal that they are not ready to start consuming from their respective partitions. # Because of quick rebalances, a consumer might have started a group, but Connect starts a rebalance, after we which we create three new instances of the consumer (one from each worker/task). But the group coordinator seems to have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Coordinator changes cause Connect integration test to fail > -- > > Key: KAFKA-7909 > URL: https://issues.apache.org/jira/browse/KAFKA-7909 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Arjun Satish >Priority: Blocker > Fix For: 2.2.0 > > > We recently introduced integration tests in Connect. This test spins up one > or more Connect workers along with a Kafka broker and Zk in a single process > and attempts to move records using a Connector. In the Example Integration > Test, we spin up three workers each hosting a Connector task that consumes > records from a Kafka topic. When the connector starts up, it may go through > multiple rounds of rebalancing. We notice the following two problems in the > last few days: > # After members join a group, there are no pendingMembers remaining, but the > join group method does not complete, and send these members a signal that > they are not ready to start consuming from their respective partitions. > # Because of quick rebalances, a consumer might have started a group, but > Connect starts a rebalance, after we which we create three new instances of > the consumer (one from each worker/task). But the group coordinator seems to > have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Even though this ticket is described in the connect of Connect, it may be > applicable to general consumers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7909) Coordinator changes cause Connect integration test to fail
Arjun Satish created KAFKA-7909: --- Summary: Coordinator changes cause Connect integration test to fail Key: KAFKA-7909 URL: https://issues.apache.org/jira/browse/KAFKA-7909 Project: Kafka Issue Type: Bug Affects Versions: 2.2.0 Reporter: Arjun Satish Fix For: 2.2.0 We recently introduced integration tests in Connect. This test spins up one or more Connect workers along with a Kafka broker and Zk in a single process and attempts to move records using a Connector. In the Example Integration Test, we spin up three workers each hosting a Connector task that consumes records from a Kafka topic. When the connector starts up, it may go through multiple rounds of rebalancing. We notice the following two problems in the last few days: # After members join a group, there are no pendingMembers remaining, but the join group method does not complete, and send these members a signal that they are not ready to start consuming from their respective partitions. # Because of quick rebalances, a consumer might have started a group, but Connect starts a rebalance, after we which we create three new instances of the consumer (one from each worker/task). But the group coordinator seems to have 4 members in the group. This causes the JoinGroup to indefinitely stall. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7877) Connect DLQ not used in SinkTask put()
[ https://issues.apache.org/jira/browse/KAFKA-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756507#comment-16756507 ] Arjun Satish commented on KAFKA-7877: - This is not a bug. We decided to not implement DLQ to capture failures in the sink task for the following reasons: # When SinkTask#put() is invoked, a collection of SinkRecords are passed into it. It is not clear which of these records, if any, are the cause of the failure. In the case of a connection error, there are no bad records per se. Either the connector needs to be resilient to such connections and retry internally or throw a RetriableException. # RetriableExceptions thrown from the SinkTask are infinitely retried (with the same set of SinkRecords). Records must make it to the DLQ only when the framework has decided if it hits a hard error, and just cannot proceed. # In some cases, the records provided to SinkTask#put() could be cached by a Connector optimized for high throughput. A later call to this method causes an error, and it might have been due to the cached records. Maybe, we should add a new Exception type (say, BadDataException) that can return the set of bad records to the framework for more precise error reporting. Could you please point to the phrasing in the KIP that led to the confusion? I'll update it so the behavior is clearer. > Connect DLQ not used in SinkTask put() > -- > > Key: KAFKA-7877 > URL: https://issues.apache.org/jira/browse/KAFKA-7877 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.0.0, 2.0.1, 2.1.0 >Reporter: Andrew Bourgeois >Priority: Major > > The Dead Letter Queue isn't implemented for the put() operation. > In > [https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java] > the "retryWithToleranceOperator" gets used during the conversion and > transformation phases, but not when delivering the messages to the sink task. > According to KIP-298, the Dead Letter Queue should be used as long as we > throw org.apache.kafka.connect.errors.RetriableException. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7772) Dynamically adjust log level in Connect workers
Arjun Satish created KAFKA-7772: --- Summary: Dynamically adjust log level in Connect workers Key: KAFKA-7772 URL: https://issues.apache.org/jira/browse/KAFKA-7772 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Arjun Satish Assignee: Arjun Satish Currently, Kafka provides a JMX interface to dynamically modify log levels of different active loggers. It would be good to have a similar interface for Connect as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7503) Integration Test Framework for Connect
[ https://issues.apache.org/jira/browse/KAFKA-7503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-7503: Summary: Integration Test Framework for Connect (was: Integration Tests for Connect) > Integration Test Framework for Connect > -- > > Key: KAFKA-7503 > URL: https://issues.apache.org/jira/browse/KAFKA-7503 > Project: Kafka > Issue Type: Task > Components: KafkaConnect >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Minor > > Implement a framework which enables writing and executing integration tests > against real connect workers and kafka brokers. The worker and brokers would > run within the same process the test is running (which is similar to how > integration tests are written in Streams and Core). The complexity of these > tests would lie somewhere between unit tests and system tests. The main > utility is to be able to run end-to-end tests within the java test framework, > and facilitate development of large features which could modify many parts of > the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7503) Integration Tests for Connect
[ https://issues.apache.org/jira/browse/KAFKA-7503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-7503: Description: Implement a framework which enables writing and executing integration tests against real connect workers and kafka brokers. The worker and brokers would run within the same process the test is running (which is similar to how integration tests are written in Streams and Core). The complexity of these tests would lie somewhere between unit tests and system tests. The main utility is to be able to run end-to-end tests within the java test framework, and facilitate development of large features which could modify many parts of the framework. > Integration Tests for Connect > - > > Key: KAFKA-7503 > URL: https://issues.apache.org/jira/browse/KAFKA-7503 > Project: Kafka > Issue Type: Task > Components: KafkaConnect >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Minor > > Implement a framework which enables writing and executing integration tests > against real connect workers and kafka brokers. The worker and brokers would > run within the same process the test is running (which is similar to how > integration tests are written in Streams and Core). The complexity of these > tests would lie somewhere between unit tests and system tests. The main > utility is to be able to run end-to-end tests within the java test framework, > and facilitate development of large features which could modify many parts of > the framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7503) Integration Tests for Connect
Arjun Satish created KAFKA-7503: --- Summary: Integration Tests for Connect Key: KAFKA-7503 URL: https://issues.apache.org/jira/browse/KAFKA-7503 Project: Kafka Issue Type: Task Components: KafkaConnect Reporter: Arjun Satish Assignee: Arjun Satish -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7228) DeadLetterQueue throws a NullPointerException
[
https://issues.apache.org/jira/browse/KAFKA-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584897#comment-16584897
]
Arjun Satish commented on KAFKA-7228:
-
Installation of hotfix jar:
# Navigate to the lib/ directory of the AK 2.0 installation.
# Replace the "connect-runtime-2.0.0.jar" file with the one provided in this
ticket: [^connect-runtime-2.0.0-HOTFIX-7228.jar] .
*Note*: this hotfix only adds the fix for the NPE described in this ticket on
the 2.0.0 release. Other bugs in connect-runtime 2.0.0 (if any) will not be
fixed by it.
> DeadLetterQueue throws a NullPointerException
> -
>
> Key: KAFKA-7228
> URL: https://issues.apache.org/jira/browse/KAFKA-7228
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Reporter: Arjun Satish
>Assignee: Arjun Satish
>Priority: Major
> Fix For: 2.0.1, 2.1.0
>
> Attachments: connect-runtime-2.0.0-HOTFIX-7228.jar
>
>
> Using the dead letter queue results in a NPE:
> {code:java}
> [2018-08-01 07:41:08,907] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task
> threw an uncaught and unrecoverable exception
> (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.report(DeadLetterQueueReporter.java:124)
> at
> org.apache.kafka.connect.runtime.errors.ProcessingContext.report(ProcessingContext.java:137)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:108)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:513)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:490)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:225)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [2018-08-01 07:41:08,908] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task is
> being killed and will not recover until manually restarted
> (org.apache.kafka.connect.runtime.WorkerTask)
> {code}
> DLQ reporter does not get a {{errorHandlingMetrics}} object when initialized
> through the WorkerSinkTask.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7228) DeadLetterQueue throws a NullPointerException
[
https://issues.apache.org/jira/browse/KAFKA-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-7228:
Attachment: connect-runtime-2.0.0-HOTFIX-7228.jar
> DeadLetterQueue throws a NullPointerException
> -
>
> Key: KAFKA-7228
> URL: https://issues.apache.org/jira/browse/KAFKA-7228
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Reporter: Arjun Satish
>Assignee: Arjun Satish
>Priority: Major
> Fix For: 2.0.1, 2.1.0
>
> Attachments: connect-runtime-2.0.0-HOTFIX-7228.jar
>
>
> Using the dead letter queue results in a NPE:
> {code:java}
> [2018-08-01 07:41:08,907] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task
> threw an uncaught and unrecoverable exception
> (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.report(DeadLetterQueueReporter.java:124)
> at
> org.apache.kafka.connect.runtime.errors.ProcessingContext.report(ProcessingContext.java:137)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:108)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:513)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:490)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:225)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [2018-08-01 07:41:08,908] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task is
> being killed and will not recover until manually restarted
> (org.apache.kafka.connect.runtime.WorkerTask)
> {code}
> DLQ reporter does not get a {{errorHandlingMetrics}} object when initialized
> through the WorkerSinkTask.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7228) DeadLetterQueue throws a NullPointerException
[
https://issues.apache.org/jira/browse/KAFKA-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-7228:
Fix Version/s: 2.1.0
2.0.1
> DeadLetterQueue throws a NullPointerException
> -
>
> Key: KAFKA-7228
> URL: https://issues.apache.org/jira/browse/KAFKA-7228
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Reporter: Arjun Satish
>Assignee: Arjun Satish
>Priority: Major
> Fix For: 2.0.1, 2.1.0
>
>
> Using the dead letter queue results in a NPE:
> {code:java}
> [2018-08-01 07:41:08,907] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task
> threw an uncaught and unrecoverable exception
> (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.report(DeadLetterQueueReporter.java:124)
> at
> org.apache.kafka.connect.runtime.errors.ProcessingContext.report(ProcessingContext.java:137)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:108)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:513)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:490)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:225)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [2018-08-01 07:41:08,908] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task is
> being killed and will not recover until manually restarted
> (org.apache.kafka.connect.runtime.WorkerTask)
> {code}
> DLQ reporter does not get a {{errorHandlingMetrics}} object when initialized
> through the WorkerSinkTask.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7228) DeadLetterQueue throws a NullPointerException
[
https://issues.apache.org/jira/browse/KAFKA-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-7228:
Description:
Using the dead letter queue results in a NPE:
{code:java}
[2018-08-01 07:41:08,907] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task
threw an uncaught and unrecoverable exception
(org.apache.kafka.connect.runtime.WorkerTask)
java.lang.NullPointerException
at
org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.report(DeadLetterQueueReporter.java:124)
at
org.apache.kafka.connect.runtime.errors.ProcessingContext.report(ProcessingContext.java:137)
at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:108)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:513)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:490)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:225)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2018-08-01 07:41:08,908] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task is
being killed and will not recover until manually restarted
(org.apache.kafka.connect.runtime.WorkerTask)
{code}
DLQ reporter does not get a {{errorHandlingMetrics}} object when initialized
through the WorkerSinkTask.
was:
Using the dead letter queue results in a NPE:
{code:java}
[2018-08-01 07:41:08,907] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task
threw an uncaught and unrecoverable exception
(org.apache.kafka.connect.runtime.WorkerTask)
java.lang.NullPointerException
at
org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.report(DeadLetterQueueReporter.java:124)
at
org.apache.kafka.connect.runtime.errors.ProcessingContext.report(ProcessingContext.java:137)
at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:108)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:513)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:490)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:225)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2018-08-01 07:41:08,908] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task is
being killed and will not recover until manually restarted
(org.apache.kafka.connect.runtime.WorkerTask)
{code}
> DeadLetterQueue throws a NullPointerException
> -
>
> Key: KAFKA-7228
> URL: https://issues.apache.org/jira/browse/KAFKA-7228
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Reporter: Arjun Satish
>Assignee: Arjun Satish
>Priority: Major
>
> Using the dead letter queue results in a NPE:
> {code:java}
> [2018-08-01 07:41:08,907] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task
> threw an uncaught and unrecoverable exception
> (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.report(DeadLetterQueueReporter.java:124)
> at
> org.apache.kafka.connect.runtime.errors.ProcessingContext.report(ProcessingContext.java:137)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:108)
> at
>
[jira] [Created] (KAFKA-7228) DeadLetterQueue throws a NullPointerException
Arjun Satish created KAFKA-7228:
---
Summary: DeadLetterQueue throws a NullPointerException
Key: KAFKA-7228
URL: https://issues.apache.org/jira/browse/KAFKA-7228
Project: Kafka
Issue Type: Task
Components: KafkaConnect
Reporter: Arjun Satish
Using the dead letter queue results in a NPE:
{code:java}
[2018-08-01 07:41:08,907] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task
threw an uncaught and unrecoverable exception
(org.apache.kafka.connect.runtime.WorkerTask)
java.lang.NullPointerException
at
org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.report(DeadLetterQueueReporter.java:124)
at
org.apache.kafka.connect.runtime.errors.ProcessingContext.report(ProcessingContext.java:137)
at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:108)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:513)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:490)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:225)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2018-08-01 07:41:08,908] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task is
being killed and will not recover until manually restarted
(org.apache.kafka.connect.runtime.WorkerTask)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7228) DeadLetterQueue throws a NullPointerException
[
https://issues.apache.org/jira/browse/KAFKA-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish reassigned KAFKA-7228:
---
Assignee: Arjun Satish
> DeadLetterQueue throws a NullPointerException
> -
>
> Key: KAFKA-7228
> URL: https://issues.apache.org/jira/browse/KAFKA-7228
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Reporter: Arjun Satish
>Assignee: Arjun Satish
>Priority: Major
>
> Using the dead letter queue results in a NPE:
> {code:java}
> [2018-08-01 07:41:08,907] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task
> threw an uncaught and unrecoverable exception
> (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.report(DeadLetterQueueReporter.java:124)
> at
> org.apache.kafka.connect.runtime.errors.ProcessingContext.report(ProcessingContext.java:137)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:108)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:513)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:490)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:321)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:225)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [2018-08-01 07:41:08,908] ERROR WorkerSinkTask{id=s3-sink-yanivr-2-4} Task is
> being killed and will not recover until manually restarted
> (org.apache.kafka.connect.runtime.WorkerTask)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7003) Add headers with error context in messages written to the Connect DeadLetterQueue topic
Arjun Satish created KAFKA-7003:
---
Summary: Add headers with error context in messages written to the
Connect DeadLetterQueue topic
Key: KAFKA-7003
URL: https://issues.apache.org/jira/browse/KAFKA-7003
Project: Kafka
Issue Type: Task
Reporter: Arjun Satish
This was added to the KIP after the feature freeze.
If the property {{errors.deadletterqueue.}}{{context.headers.enable}} is set to
{{*true*}}, the following headers will be added to the produced raw message
(only if they don't already exist in the message). All values will be
serialized as UTF-8 strings.
||Header Name||Description||
|__connect.errors.topic|Name of the topic that contained the message.|
|__connect.errors.task.id|The numeric ID of the task that encountered the error
(encoded as a UTF-8 string).|
|__connect.errors.stage|The name of the stage where the error occurred.|
|__connect.errors.partition|The numeric ID of the partition in the original
topic that contained the message (encoded as a UTF-8 string).|
|__connect.errors.offset|The numeric value of the message offset in the
original topic (encoded as a UTF-8 string).|
|__connect.errors.exception.stacktrace|The stacktrace of the exception.|
|__connect.errors.exception.message|The message in the exception.|
|__connect.errors.exception.class.name|The fully qualified classname of the
exception that was thrown during the execution.|
|__connect.errors.connector.name|The name of the connector which encountered
the error.|
|__connect.errors.class.name|The fully qualified name of the class that caused
the error.|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7002) Allow replication factor to be set via a configuration property for the Connect DLQ topic
Arjun Satish created KAFKA-7002: --- Summary: Allow replication factor to be set via a configuration property for the Connect DLQ topic Key: KAFKA-7002 URL: https://issues.apache.org/jira/browse/KAFKA-7002 Project: Kafka Issue Type: Task Components: KafkaConnect Affects Versions: 2.0.0 Reporter: Arjun Satish Assignee: Arjun Satish Fix For: 2.0.0 Currently, the replication factor is hardcoded to a value of 3. This means that we cannot use a DLQ in any cluster setup with less than three brokers. It is better to have the user specify this value if the default value does meet the requirements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7001) Rename errors.allowed.max property in Connect to errors.tolerance
[ https://issues.apache.org/jira/browse/KAFKA-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-7001: Summary: Rename errors.allowed.max property in Connect to errors.tolerance (was: Rename `errors.allowed.max` in Connect to `errors.tolerance`) > Rename errors.allowed.max property in Connect to errors.tolerance > - > > Key: KAFKA-7001 > URL: https://issues.apache.org/jira/browse/KAFKA-7001 > Project: Kafka > Issue Type: Task > Components: KafkaConnect >Affects Versions: 2.0.0 >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Minor > Fix For: 2.0.0 > > > Since errors.allowed.max indicates a continuous value of possibilities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7001) Rename `errors.allowed.max` in Connect to `errors.tolerance`
[ https://issues.apache.org/jira/browse/KAFKA-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-7001: Description: Since errors.allowed.max indicates a continuous value of possibilities > Rename `errors.allowed.max` in Connect to `errors.tolerance` > > > Key: KAFKA-7001 > URL: https://issues.apache.org/jira/browse/KAFKA-7001 > Project: Kafka > Issue Type: Task > Components: KafkaConnect >Affects Versions: 2.0.0 >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Minor > Fix For: 2.0.0 > > > Since errors.allowed.max indicates a continuous value of possibilities -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7001) Rename `errors.allowed.max` in Connect to `errors.tolerance`
Arjun Satish created KAFKA-7001: --- Summary: Rename `errors.allowed.max` in Connect to `errors.tolerance` Key: KAFKA-7001 URL: https://issues.apache.org/jira/browse/KAFKA-7001 Project: Kafka Issue Type: Task Components: KafkaConnect Affects Versions: 2.0.0 Reporter: Arjun Satish Assignee: Arjun Satish Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6981) Missing Connector Config (errors.deadletterqueue.topic.name) kills Connect Clusters
[ https://issues.apache.org/jira/browse/KAFKA-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-6981: Description: The trunk version of AK currently tries to incorrectly read the property (errors.deadletterqueue.topic.name) when starting a sink connector. This fails no matter what the contents of the connector config are. The ConnectorConfig does not define this property, and any calls to getString will throw a ConfigException (since only known properties are retained by AbstractConfig). (was: The trunk version of AK currently tries to incorrectly read the property (errors.deadletterqueue.topic.name) when starting a sink connector. This fails no matter what the contents of the connector config are. The ConnectorConfig does not define this property, and any calls to getString will throw a ConfigException. ) > Missing Connector Config (errors.deadletterqueue.topic.name) kills Connect > Clusters > --- > > Key: KAFKA-6981 > URL: https://issues.apache.org/jira/browse/KAFKA-6981 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Major > Fix For: 2.0.0 > > > The trunk version of AK currently tries to incorrectly read the property > (errors.deadletterqueue.topic.name) when starting a sink connector. This > fails no matter what the contents of the connector config are. The > ConnectorConfig does not define this property, and any calls to getString > will throw a ConfigException (since only known properties are retained by > AbstractConfig). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6981) Missing Connector Config (errors.deadletterqueue.topic.name) kills Connect Clusters
[ https://issues.apache.org/jira/browse/KAFKA-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-6981: Priority: Major (was: Minor) > Missing Connector Config (errors.deadletterqueue.topic.name) kills Connect > Clusters > --- > > Key: KAFKA-6981 > URL: https://issues.apache.org/jira/browse/KAFKA-6981 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Major > Fix For: 2.0.0 > > > The trunk version of AK currently tries to incorrectly read the property > (errors.deadletterqueue.topic.name) when starting a sink connector. This > fails no matter what the contents of the connector config are. The > ConnectorConfig does not define this property, and any calls to getString > will throw a ConfigException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6981) Missing Connector Config (errors.deadletterqueue.topic.name) kills Connect Clusters
Arjun Satish created KAFKA-6981: --- Summary: Missing Connector Config (errors.deadletterqueue.topic.name) kills Connect Clusters Key: KAFKA-6981 URL: https://issues.apache.org/jira/browse/KAFKA-6981 Project: Kafka Issue Type: Bug Affects Versions: 2.0.0 Reporter: Arjun Satish Assignee: Arjun Satish Fix For: 2.0.0 The trunk version of AK currently tries to incorrectly read the property (errors.deadletterqueue.topic.name) when starting a sink connector. This fails no matter what the contents of the connector config are. The ConnectorConfig does not define this property, and any calls to getString will throw a ConfigException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-6833) KafkaProducer throws "Invalid partition given with record" exception
[
https://issues.apache.org/jira/browse/KAFKA-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-6833:
Description:
Currently, when creating topics via ZooKeeper, there is a small but definite
delay between creating the nodes in ZK, and having the topics created in the
brokers. the KafkaProducer maintains a metadata cache about topics which get
updated after the broker metadata is updated. If an application adds partitions
to a topic, and immediately tries to produce records to a new partition, a
KafkaException is throw with a message similar to the following:
{code:java}
Caused by: org.apache.kafka.common.KafkaException: Invalid partition given with
record: 12 is not in the range [0...1).
{code}
In this case, since the application has context that it created the topics, it
might be worthwhile to consider if a more specific exception can be thrown
instead of KafkaException. For example:
{code:java}
public class PartitionNotFoundException extends KafkaException {...}{code}
This could allow the application to be able to interpret such an error, and act
accordingly.
EDIT: Correct "create topics" to "adds partitions to a topic".
was:
Currently, when creating topics via ZooKeeper, there is a small but definite
delay between creating the nodes in ZK, and having the topics created in the
brokers. the KafkaProducer maintains a metadata cache about topics which get
updated after the broker metadata is updated. If an application adds partitions
to a topic, and immediately tries to produce records to a new partition, a
KafkaException is throw with a message similar to the following:
{code:java}
Caused by: org.apache.kafka.common.KafkaException: Invalid partition given with
record: 12 is not in the range [0...1).
{code}
In this case, since the application has context that it created the topics, it
might be worthwhile to consider if a more specific exception can be thrown
instead of KafkaException. For example:
{code:java}
public class PartitionNotFoundException extends KafkaException {...}{code}
This could allow the application to be able to interpret such an error, and act
accordingly.
EDIT: Update "create topics" to "adds partitions to a topic"
> KafkaProducer throws "Invalid partition given with record" exception
>
>
> Key: KAFKA-6833
> URL: https://issues.apache.org/jira/browse/KAFKA-6833
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Arjun Satish
>Priority: Minor
>
> Currently, when creating topics via ZooKeeper, there is a small but definite
> delay between creating the nodes in ZK, and having the topics created in the
> brokers. the KafkaProducer maintains a metadata cache about topics which get
> updated after the broker metadata is updated. If an application adds
> partitions to a topic, and immediately tries to produce records to a new
> partition, a KafkaException is throw with a message similar to the following:
> {code:java}
> Caused by: org.apache.kafka.common.KafkaException: Invalid partition given
> with record: 12 is not in the range [0...1).
> {code}
> In this case, since the application has context that it created the topics,
> it might be worthwhile to consider if a more specific exception can be thrown
> instead of KafkaException. For example:
> {code:java}
> public class PartitionNotFoundException extends KafkaException {...}{code}
> This could allow the application to be able to interpret such an error, and
> act accordingly.
> EDIT: Correct "create topics" to "adds partitions to a topic".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-6833) KafkaProducer throws "Invalid partition given with record" exception
[
https://issues.apache.org/jira/browse/KAFKA-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-6833:
Description:
Currently, when creating topics via ZooKeeper, there is a small but definite
delay between creating the nodes in ZK, and having the topics created in the
brokers. the KafkaProducer maintains a metadata cache about topics which get
updated after the broker metadata is updated. If an application adds partitions
to a topic, and immediately tries to produce records to a new partition, a
KafkaException is throw with a message similar to the following:
{code:java}
Caused by: org.apache.kafka.common.KafkaException: Invalid partition given with
record: 12 is not in the range [0...1).
{code}
In this case, since the application has context that it created the topics, it
might be worthwhile to consider if a more specific exception can be thrown
instead of KafkaException. For example:
{code:java}
public class PartitionNotFoundException extends KafkaException {...}{code}
This could allow the application to be able to interpret such an error, and act
accordingly.
EDIT: Update "create topics" to "adds partitions to a topic"
was:
Currently, when creating topics via ZooKeeper, there is a small but definite
delay between creating the nodes in ZK, and having the topics created in the
brokers. the KafkaProducer maintains a metadata cache about topics which get
updated after the broker metadata is updated. If an application adds partitions
to a topic, and immediately tries to produce records to a new partition, a
KafkaException is throw with a message similar to the following:
{code:java}
Caused by: org.apache.kafka.common.KafkaException: Invalid partition given with
record: 12 is not in the range [0...1).
{code}
In this case, since the application has context that it created the topics, it
might be worthwhile to consider if a more specific exception can be thrown
instead of KafkaException. For example:
{code:java}
public class PartitionNotFoundException extends KafkaException {...}{code}
This could allow the application to be able to interpret such an error, and act
accordingly.
> KafkaProducer throws "Invalid partition given with record" exception
>
>
> Key: KAFKA-6833
> URL: https://issues.apache.org/jira/browse/KAFKA-6833
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Arjun Satish
>Priority: Minor
>
> Currently, when creating topics via ZooKeeper, there is a small but definite
> delay between creating the nodes in ZK, and having the topics created in the
> brokers. the KafkaProducer maintains a metadata cache about topics which get
> updated after the broker metadata is updated. If an application adds
> partitions to a topic, and immediately tries to produce records to a new
> partition, a KafkaException is throw with a message similar to the following:
> {code:java}
> Caused by: org.apache.kafka.common.KafkaException: Invalid partition given
> with record: 12 is not in the range [0...1).
> {code}
> In this case, since the application has context that it created the topics,
> it might be worthwhile to consider if a more specific exception can be thrown
> instead of KafkaException. For example:
> {code:java}
> public class PartitionNotFoundException extends KafkaException {...}{code}
> This could allow the application to be able to interpret such an error, and
> act accordingly.
> EDIT: Update "create topics" to "adds partitions to a topic"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-6833) KafkaProducer throws "Invalid partition given with record" exception
[
https://issues.apache.org/jira/browse/KAFKA-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-6833:
Description:
Currently, when creating topics via ZooKeeper, there is a small but definite
delay between creating the nodes in ZK, and having the topics created in the
brokers. the KafkaProducer maintains a metadata cache about topics which get
updated after the broker metadata is updated. If an application adds partitions
to a topic, and immediately tries to produce records to a new partition, a
KafkaException is throw with a message similar to the following:
{code:java}
Caused by: org.apache.kafka.common.KafkaException: Invalid partition given with
record: 12 is not in the range [0...1).
{code}
In this case, since the application has context that it created the topics, it
might be worthwhile to consider if a more specific exception can be thrown
instead of KafkaException. For example:
{code:java}
public class PartitionNotFoundException extends KafkaException {...}{code}
This could allow the application to be able to interpret such an error, and act
accordingly.
was:
Currently, when creating topics via ZooKeeper, there is a small but definite
delay between creating the nodes in ZK, and having the topics created in the
brokers. the KafkaProducer maintains a metadata cache about topics which get
updated after the broker metadata is updated. If an application creates topics,
and immediately tries to produce records to a new partition, a KafkaException
is throw with a message similar to the following:
{code:java}
Caused by: org.apache.kafka.common.KafkaException: Invalid partition given with
record: 12 is not in the range [0...1).
{code}
In this case, since the application has context that it created the topics, it
might be worthwhile to consider if a more specific exception can be thrown
instead of KafkaException. For example:
{code:java}
public class PartitionNotFoundException extends KafkaException {...}{code}
This could allow the application to be able to interpret such an error, and act
accordingly.
> KafkaProducer throws "Invalid partition given with record" exception
>
>
> Key: KAFKA-6833
> URL: https://issues.apache.org/jira/browse/KAFKA-6833
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Arjun Satish
>Priority: Minor
>
> Currently, when creating topics via ZooKeeper, there is a small but definite
> delay between creating the nodes in ZK, and having the topics created in the
> brokers. the KafkaProducer maintains a metadata cache about topics which get
> updated after the broker metadata is updated. If an application adds
> partitions to a topic, and immediately tries to produce records to a new
> partition, a KafkaException is throw with a message similar to the following:
> {code:java}
> Caused by: org.apache.kafka.common.KafkaException: Invalid partition given
> with record: 12 is not in the range [0...1).
> {code}
> In this case, since the application has context that it created the topics,
> it might be worthwhile to consider if a more specific exception can be thrown
> instead of KafkaException. For example:
> {code:java}
> public class PartitionNotFoundException extends KafkaException {...}{code}
> This could allow the application to be able to interpret such an error, and
> act accordingly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-6833) KafkaProducer throws "Invalid partition given with record" exception
Arjun Satish created KAFKA-6833:
---
Summary: KafkaProducer throws "Invalid partition given with
record" exception
Key: KAFKA-6833
URL: https://issues.apache.org/jira/browse/KAFKA-6833
Project: Kafka
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Arjun Satish
Currently, when creating topics via ZooKeeper, there is a small but definite
delay between creating the nodes in ZK, and having the topics created in the
brokers. the KafkaProducer maintains a metadata cache about topics which get
updated after the broker metadata is updated. If an application creates topics,
and immediately tries to produce records to a new partition, a KafkaException
is throw with a message similar to the following:
{code:java}
Caused by: org.apache.kafka.common.KafkaException: Invalid partition given with
record: 12 is not in the range [0...1).
{code}
In this case, since the application has context that it created the topics, it
might be worthwhile to consider if a more specific exception can be thrown
instead of KafkaException. For example:
{code:java}
public class PartitionNotFoundException extends KafkaException {...}{code}
This could allow the application to be able to interpret such an error, and act
accordingly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-6511) Connect header parser incorrectly parses arrays
Arjun Satish created KAFKA-6511: --- Summary: Connect header parser incorrectly parses arrays Key: KAFKA-6511 URL: https://issues.apache.org/jira/browse/KAFKA-6511 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 1.1.0 Reporter: Arjun Satish Assignee: Randall Hauch An incorrect input like "[1, 2, 3,,,]" is misinterpreted by the Values parser. An example test can be found here: https://github.com/apache/kafka/pull/4319#discussion_r165155768 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KAFKA-4827) Kafka connect: error with special characters in connector name
[
https://issues.apache.org/jira/browse/KAFKA-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish closed KAFKA-4827.
---
Changes have been merged and backported to 0.10.0, 0.10.1, 0.10.2 and 0.11.0
branches.
> Kafka connect: error with special characters in connector name
> --
>
> Key: KAFKA-4827
> URL: https://issues.apache.org/jira/browse/KAFKA-4827
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.1.0
>Reporter: Aymeric Bouvet
>Assignee: Arjun Satish
>Priority: Minor
> Fix For: 0.10.0.2, 0.10.1.2, 0.10.2.2, 1.1.0, 1.0.1, 0.11.0.3
>
>
> When creating a connector, if the connector name (and possibly other
> properties) end with a carriage return, kafka-connect will create the config
> but report error
> {code}
> cat << EOF > file-connector.json
> {
> "name": "file-connector\r",
> "config": {
> "topic": "kafka-connect-logs\r",
> "tasks.max": "1",
> "file": "/var/log/ansible-confluent/connect.log",
> "connector.class":
> "org.apache.kafka.connect.file.FileStreamSourceConnector"
> }
> }
> EOF
> curl -X POST -H "Content-Type: application/json" -H "Accept:
> application/json" -d @file-connector.json localhost:8083/connectors
> {code}
> returns an error 500 and log the following
> {code}
> [2017-03-01 18:25:23,895] WARN (org.eclipse.jetty.servlet.ServletHandler)
> javax.servlet.ServletException: java.lang.IllegalArgumentException: Illegal
> character in path at index 27: /connectors/file-connector4
> at
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
> at
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
> at
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at
> org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Illegal character in path at
> index 27: /connectors/file-connector4
> at java.net.URI.create(URI.java:852)
> at
> org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.createConnector(ConnectorsResource.java:100)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
> at
> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
> at
>
[jira] [Commented] (KAFKA-6193) ReassignPartitionsClusterTest.shouldPerformMultipleReassignmentOperationsOverVariousTopics fails sometimes
[
https://issues.apache.org/jira/browse/KAFKA-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255688#comment-16255688
]
Arjun Satish commented on KAFKA-6193:
-
This test is failing for a PR I created. The Jenkins build is
[here|https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/9469/testReport/kafka.admin/ReassignPartitionsClusterTest/shouldPerformMultipleReassignmentOperationsOverVariousTopics/].
> ReassignPartitionsClusterTest.shouldPerformMultipleReassignmentOperationsOverVariousTopics
> fails sometimes
> --
>
> Key: KAFKA-6193
> URL: https://issues.apache.org/jira/browse/KAFKA-6193
> Project: Kafka
> Issue Type: Test
>Reporter: Ted Yu
> Fix For: 1.1.0, 1.0.1
>
> Attachments: 6193.out
>
>
> From
> https://builds.apache.org/job/kafka-trunk-jdk8/2198/testReport/junit/kafka.admin/ReassignPartitionsClusterTest/shouldPerformMultipleReassignmentOperationsOverVariousTopics/
> :
> {code}
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> kafka.admin.ReassignPartitionsClusterTest.shouldPerformMultipleReassignmentOperationsOverVariousTopics(ReassignPartitionsClusterTest.scala:524)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (KAFKA-4115) Grow default heap settings for distributed Connect from 256M to 1G
[
https://issues.apache.org/jira/browse/KAFKA-4115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish reassigned KAFKA-4115:
---
Assignee: Arjun Satish (was: Shikhar Bhushan)
> Grow default heap settings for distributed Connect from 256M to 1G
> --
>
> Key: KAFKA-4115
> URL: https://issues.apache.org/jira/browse/KAFKA-4115
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Shikhar Bhushan
>Assignee: Arjun Satish
> Labels: newbie
>
> Currently, both {{connect-standalone.sh}} and {{connect-distributed.sh}}
> start the Connect JVM with the default heap settings from
> {{kafka-run-class.sh}} of {{-Xmx256M}}.
> At least for distributed connect, we should default to a much higher limit
> like 1G. While the 'correct' sizing is workload dependent, with a system
> where you can run arbitrary connector plugins which may perform buffering of
> data, we should provide for more headroom.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
