[jira] [Commented] (KAFKA-9376) Plugin class loader not found using MM2
[
https://issues.apache.org/jira/browse/KAFKA-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039776#comment-17039776
]
candan commented on KAFKA-9376:
---
Hi;
Acctually, I didn't set any {{plugin.path. Just open instal tar under a
directory. By the way, mirror maker2 is working. }}
> Plugin class loader not found using MM2
> ---
>
> Key: KAFKA-9376
> URL: https://issues.apache.org/jira/browse/KAFKA-9376
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.4.0
>Reporter: Sinóros-Szabó Péter
>Priority: Minor
>
> I am using MM2 (release 2.4.0 with scala 2.12) I geta bunch of classloader
> errors. MM2 seems to be working, but I do not know if all of it components
> are working as expected as this is the first time I use MM2.
> I run MM2 with the following command:
> {code:java}
> ./bin/connect-mirror-maker.sh config/connect-mirror-maker.properties
> {code}
> Errors are:
> {code:java}
> [2020-01-07 15:06:17,892] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,889] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,904] INFO ConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
> (org.apache.kafka.connect.runtime.ConnectorConfig:347)
> [2020-01-07 15:06:17,904] INFO EnrichedConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
>
> (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
> [2020-01-07 15:06:17,905] INFO TaskConfig values:
> task.class = class org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.TaskConfig:347)
> [2020-01-07 15:06:17,905] INFO Instantiated task MirrorHeartbeatConnector-0
> with version 1 of type org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.Worker:434){code}
> After a while, these errors are not logged any more.
> Config is:
> {code:java}
> clusters = eucmain, euwbackup
> eucmain.bootstrap.servers = kafka1:9092,kafka2:9092
> euwbackup.bootstrap.servers = 172.30.197.203:9092,172.30.213.104:9092
> eucmain->euwbackup.enabled = true
> eucmain->euwbackup.topics = .*
> eucmain->euwbackup.topics.blacklist = ^(kafka|kmf|__|pricing).*
> eucmain->euwbackup.rename.topics = false
> rename.topics = false
> eucmain->euwbackup.sync.topic.acls.enabled = false
> sync.topic.acls.enabled = false{code}
> Using OpenJDK 8 or 11, I get the same error.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9571) MirrorMaker task failing during pool
[
https://issues.apache.org/jira/browse/KAFKA-9571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nitish Goyal updated KAFKA-9571:
Description:
I have setup kafka replication between source and target cluster
I am observing Mirror Source task getting killed after certain time with the
following error
```
[[2020-02-17 22:39:57,344] ERROR Failure during poll.
(org.apache.kafka.connect.mirror.MirrorSourceTask:161)
[2020-02-17 22:39:57,346] ERROR WorkerSourceTask\{id=MirrorSourceConnector-99}
Task threw an uncaught and unrecoverable exception
(org.apache.kafka.connect.runtime.WorkerTask:179)
[2020-02-17 22:39:57,346] ERROR WorkerSourceTask\{id=MirrorSourceConnector-99}
Task is being killed and will not recover until manually restarted
(org.apache.kafka.connect.runtime.WorkerTask:180)
```
What could be the possible reason for the above?
was:
I have setup kafka replication between source and target cluster
I am observing Mirror Source task getting killed after certain time with the
following error
```
[[2020-02-17 22:39:57,344] ERROR Failure during poll.
(org.apache.kafka.connect.mirror.MirrorSourceTask:161)
[2020-02-17 22:39:57,346] ERROR WorkerSourceTask\{id=MirrorSourceConnector-99}
Task threw an uncaught and unrecoverable exception
(org.apache.kafka.connect.runtime.WorkerTask:179)
[2020-02-17 22:39:57,346] ERROR WorkerSourceTask\{id=MirrorSourceConnector-99}
Task is being killed and will not recover until manually restarted
(org.apache.kafka.connect.runtime.WorkerTask:180)
```
What could be the possible reason for the above?
> MirrorMaker task failing during pool
>
>
> Key: KAFKA-9571
> URL: https://issues.apache.org/jira/browse/KAFKA-9571
> Project: Kafka
> Issue Type: Bug
> Components: consumer, mirrormaker
>Affects Versions: 2.4.0
>Reporter: Nitish Goyal
>Priority: Blocker
>
> I have setup kafka replication between source and target cluster
> I am observing Mirror Source task getting killed after certain time with the
> following error
>
> ```
> [[2020-02-17 22:39:57,344] ERROR Failure during poll.
> (org.apache.kafka.connect.mirror.MirrorSourceTask:161)
> [2020-02-17 22:39:57,346] ERROR
> WorkerSourceTask\{id=MirrorSourceConnector-99} Task threw an uncaught and
> unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)
> [2020-02-17 22:39:57,346] ERROR
> WorkerSourceTask\{id=MirrorSourceConnector-99} Task is being killed and will
> not recover until manually restarted
> (org.apache.kafka.connect.runtime.WorkerTask:180)
> ```
>
> What could be the possible reason for the above?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9571) MirrorMaker task failing during pool
Nitish Goyal created KAFKA-9571:
---
Summary: MirrorMaker task failing during pool
Key: KAFKA-9571
URL: https://issues.apache.org/jira/browse/KAFKA-9571
Project: Kafka
Issue Type: Bug
Components: consumer, mirrormaker
Affects Versions: 2.4.0
Reporter: Nitish Goyal
I have setup kafka replication between source and target cluster
I am observing Mirror Source task getting killed after certain time with the
following error
```
[[2020-02-17 22:39:57,344] ERROR Failure during poll.
(org.apache.kafka.connect.mirror.MirrorSourceTask:161)
[2020-02-17 22:39:57,346] ERROR WorkerSourceTask\{id=MirrorSourceConnector-99}
Task threw an uncaught and unrecoverable exception
(org.apache.kafka.connect.runtime.WorkerTask:179)
[2020-02-17 22:39:57,346] ERROR WorkerSourceTask\{id=MirrorSourceConnector-99}
Task is being killed and will not recover until manually restarted
(org.apache.kafka.connect.runtime.WorkerTask:180)
```
What could be the possible reason for the above?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9568) Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership
[ https://issues.apache.org/jira/browse/KAFKA-9568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David J. Garcia updated KAFKA-9568: --- Affects Version/s: (was: 2.3.0) 2.4.0 > Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership > > > Key: KAFKA-9568 > URL: https://issues.apache.org/jira/browse/KAFKA-9568 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.4.0 >Reporter: David J. Garcia >Priority: Major > > A kstreams application with static membership, and > StreamsConfg.APPLICATION_SERVER_CONFIG set, will NOT update old server config > upon restart of application on new host. > Steps to reproduce: > > # start two kstreams applications (with same consumer group) and enable > static membership (and set application server config to :) > # kill one of the applications and restart it on a new host(with new ip) > before timeout ends (so that rebalancing doesn't occur). > # the other kstreams application will now have an invalid > application_server_config > Possible fix: > If an application restarts with a new host/identity..etc, it could trigger a > "light-rebalance" where the other applications in the consumer group don't > change partition assignments ,but instead just get their configuration > updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9568) Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership
[
https://issues.apache.org/jira/browse/KAFKA-9568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039685#comment-17039685
]
David J. Garcia commented on KAFKA-9568:
B would have an incorrect A mapping. For example, suppose that there was a
state-store with some data on A and other data on B. Here is some scala that
I'm using:
{code:java}
val keyHostInfo: Option[IQHost] = Option(streamsRef.metadataForKey(
storeName,
docId,
Implicits.keySerde.serializer)).map(m => IQHost(m.host(), m.port(),
thisHost.IQConfig))
//m is of type StreamsMetadata
{code}
StreamsMetadata has mappings that link the partition of docId to a host/port.
Sometimes the host is the current machine, other times its on another machine
(as reported by that machine to the controller)
> Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership
>
>
> Key: KAFKA-9568
> URL: https://issues.apache.org/jira/browse/KAFKA-9568
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.3.0
>Reporter: David J. Garcia
>Priority: Major
>
> A kstreams application with static membership, and
> StreamsConfg.APPLICATION_SERVER_CONFIG set, will NOT update old server config
> upon restart of application on new host.
> Steps to reproduce:
>
> # start two kstreams applications (with same consumer group) and enable
> static membership (and set application server config to :)
> # kill one of the applications and restart it on a new host(with new ip)
> before timeout ends (so that rebalancing doesn't occur).
> # the other kstreams application will now have an invalid
> application_server_config
> Possible fix:
> If an application restarts with a new host/identity..etc, it could trigger a
> "light-rebalance" where the other applications in the consumer group don't
> change partition assignments ,but instead just get their configuration
> updated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9142) topic __transaction_state disk space grows too big
[
https://issues.apache.org/jira/browse/KAFKA-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039623#comment-17039623
]
Boquan Tang commented on KAFKA-9142:
Is this the similar to https://issues.apache.org/jira/browse/KAFKA-8335 ? The
issue is resolved after 2.2.1 so you may want to update your broker.
> topic __transaction_state disk space grows too big
> --
>
> Key: KAFKA-9142
> URL: https://issues.apache.org/jira/browse/KAFKA-9142
> Project: Kafka
> Issue Type: Improvement
> Components: log cleaner
>Affects Versions: 2.0.0
>Reporter: panpan.liu
>Priority: Critical
> Attachments: image-2019-11-05-15-31-46-859.png,
> image-2019-11-05-15-33-55-632.png, image-2019-11-05-15-36-52-235.png
>
>
> kafka broker :2.0.0
> kafka stream client: 2.1.0
> topic description please see attachements.
> The broker runs from 10/2018. I find the topic grows up and does not delete
> (*cleanup.policy=compact*).
> How can I handle it ?
>
> {code:java}
> # topic description(default)
> Topic:__transaction_state PartitionCount:50 ReplicationFactor:2
> Configs:segment.bytes=104857600,unclean.leader.election.enable=false,compression.type=uncompressed,cleanup.policy=compact,min.insync.replicas=1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9440) Add ConsumerGroupCommand to delete static members
[ https://issues.apache.org/jira/browse/KAFKA-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039606#comment-17039606 ] Xue Liu commented on KAFKA-9440: Hi [~hachikuji] can you help to add me the permission to create KIP? I probably also need the permission to assign ownership. Tried to ping the dev email yet no reply. Thanks! id: xuel1 > Add ConsumerGroupCommand to delete static members > - > > Key: KAFKA-9440 > URL: https://issues.apache.org/jira/browse/KAFKA-9440 > Project: Kafka > Issue Type: Improvement >Reporter: Boyang Chen >Priority: Major > Labels: help-wanted, kip, newbie, newbie++ > > We introduced a new AdminClient API removeMembersFromConsumerGroup in 2.4. It > would be good to instantiate the API as part of the ConsumerGroupCommand for > easy command line usage. > This change requires a new KIP, and just posting out here in case anyone who > uses static membership to pick it up, if they would like to use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9306) Kafka Consumer does not clean up all metrics after shutdown
[ https://issues.apache.org/jira/browse/KAFKA-9306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjana Kaundinya resolved KAFKA-9306. -- Fix Version/s: 2.4.1 Resolution: Fixed > Kafka Consumer does not clean up all metrics after shutdown > --- > > Key: KAFKA-9306 > URL: https://issues.apache.org/jira/browse/KAFKA-9306 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Major > Fix For: 2.4.1 > > > The Kafka Consumer does not clean up all metrics after shutdown. It seems > like this was a regression introduced in Kafka 2.4 when we added the > KafkaConsumerMetrics class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9439) Add more public API tests for KafkaProducer
[ https://issues.apache.org/jira/browse/KAFKA-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039588#comment-17039588 ] Boyang Chen commented on KAFKA-9439: [~iamabug] Sorry for late reply, this ticket seems to be taken by someone else. Could you send an email [to d...@kafka.apache.org|mailto:to%c2%a0...@kafka.apache.org] to get Jira access first? Once you got that, feel free to ping me and I could find some starter works for you, thanks! > Add more public API tests for KafkaProducer > --- > > Key: KAFKA-9439 > URL: https://issues.apache.org/jira/browse/KAFKA-9439 > Project: Kafka > Issue Type: Improvement >Reporter: Boyang Chen >Assignee: Jeff Kim >Priority: Major > Labels: newbie > > While working on KIP-447, we realized a lack of test coverage on the > KafkaProducer public APIs. For example, `commitTransaction` and > `sendOffsetsToTransaction` are not even called in the > `KafkaProducerTest.java` and the code coverage is only 75%. > Adding more unit tests here will be pretty valuable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-6266) Kafka 1.0.0 : Repeated occurrence of WARN Resetting first dirty offset of __consumer_offsets-xx to log start offset 203569 since the checkpointed offset 120955 is inval
[
https://issues.apache.org/jira/browse/KAFKA-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039568#comment-17039568
]
ASF GitHub Bot commented on KAFKA-6266:
---
splett2 commented on pull request #8136: KAFKA-6266: Repeated occurrence of
WARN Resetting first dirty offset …
URL: https://github.com/apache/kafka/pull/8136
…(#8089)
Previously, checkpointed offsets for a log were only updated if the log was
chosen for cleaning once the cleaning job completes. This caused issues in
cases where logs with invalid checkpointed offsets would repeatedly emit
warnings if the log with an invalid cleaning checkpoint wasn't chosen for
cleaning.
Proposed fix is to update the checkpointed offset for logs with invalid
checkpoints regardless of whether it gets chosen for cleaning.
Reviewers: Anna Povzner , Jun Rao
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Kafka 1.0.0 : Repeated occurrence of WARN Resetting first dirty offset of
> __consumer_offsets-xx to log start offset 203569 since the checkpointed
> offset 120955 is invalid. (kafka.log.LogCleanerManager$)
> --
>
> Key: KAFKA-6266
> URL: https://issues.apache.org/jira/browse/KAFKA-6266
> Project: Kafka
> Issue Type: Bug
> Components: offset manager
>Affects Versions: 1.0.0, 1.0.1
> Environment: CentOS 7, Apache kafka_2.12-1.0.0
>Reporter: VinayKumar
>Assignee: David Mao
>Priority: Major
> Fix For: 2.5.0, 2.4.1
>
>
> I upgraded Kafka from 0.10.2.1 to 1.0.0 version. From then, I see the below
> warnings in the log.
> I'm seeing these continuously in the log, and want these to be fixed- so
> that they wont repeat. Can someone please help me in fixing the below
> warnings.
> {code}
> WARN Resetting first dirty offset of __consumer_offsets-17 to log start
> offset 3346 since the checkpointed offset 3332 is invalid.
> (kafka.log.LogCleanerManager$)
> WARN Resetting first dirty offset of __consumer_offsets-23 to log start
> offset 4 since the checkpointed offset 1 is invalid.
> (kafka.log.LogCleanerManager$)
> WARN Resetting first dirty offset of __consumer_offsets-19 to log start
> offset 203569 since the checkpointed offset 120955 is invalid.
> (kafka.log.LogCleanerManager$)
> WARN Resetting first dirty offset of __consumer_offsets-35 to log start
> offset 16957 since the checkpointed offset 7 is invalid.
> (kafka.log.LogCleanerManager$)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-9439) Add more public API tests for KafkaProducer
[ https://issues.apache.org/jira/browse/KAFKA-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Kim reassigned KAFKA-9439: --- Assignee: Jeff Kim > Add more public API tests for KafkaProducer > --- > > Key: KAFKA-9439 > URL: https://issues.apache.org/jira/browse/KAFKA-9439 > Project: Kafka > Issue Type: Improvement >Reporter: Boyang Chen >Assignee: Jeff Kim >Priority: Major > Labels: newbie > > While working on KIP-447, we realized a lack of test coverage on the > KafkaProducer public APIs. For example, `commitTransaction` and > `sendOffsetsToTransaction` are not even called in the > `KafkaProducerTest.java` and the code coverage is only 75%. > Adding more unit tests here will be pretty valuable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions
[
https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039548#comment-17039548
]
Harsha commented on KAFKA-4084:
---
[~blodsbror] if you are interested in trying out our patch we are happy to
provide on top of whichever Kafka version you are testing.
If you can provide your test results by using this patch it gives further more
evidence for this KIP. Let us know.
> automated leader rebalance causes replication downtime for clusters with too
> many partitions
>
>
> Key: KAFKA-4084
> URL: https://issues.apache.org/jira/browse/KAFKA-4084
> Project: Kafka
> Issue Type: Bug
> Components: controller
>Affects Versions: 0.8.2.2, 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Tom Crayford
>Priority: Major
> Labels: reliability
> Fix For: 1.1.0
>
>
> If you enable {{auto.leader.rebalance.enable}} (which is on by default), and
> you have a cluster with many partitions, there is a severe amount of
> replication downtime following a restart. This causes
> `UnderReplicatedPartitions` to fire, and replication is paused.
> This is because the current automated leader rebalance mechanism changes
> leaders for *all* imbalanced partitions at once, instead of doing it
> gradually. This effectively stops all replica fetchers in the cluster
> (assuming there are enough imbalanced partitions), and restarts them. This
> can take minutes on busy clusters, during which no replication is happening
> and user data is at risk. Clients with {{acks=-1}} also see issues at this
> time, because replication is effectively stalled.
> To quote Todd Palino from the mailing list:
> bq. There is an admin CLI command to trigger the preferred replica election
> manually. There is also a broker configuration “auto.leader.rebalance.enable”
> which you can set to have the broker automatically perform the PLE when
> needed. DO NOT USE THIS OPTION. There are serious performance issues when
> doing so, especially on larger clusters. It needs some development work that
> has not been fully identified yet.
> This setting is extremely useful for smaller clusters, but with high
> partition counts causes the huge issues stated above.
> One potential fix could be adding a new configuration for the number of
> partitions to do automated leader rebalancing for at once, and *stop* once
> that number of leader rebalances are in flight, until they're done. There may
> be better mechanisms, and I'd love to hear if anybody has any ideas.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-6266) Kafka 1.0.0 : Repeated occurrence of WARN Resetting first dirty offset of __consumer_offsets-xx to log start offset 203569 since the checkpointed offset 120955 is inval
[
https://issues.apache.org/jira/browse/KAFKA-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039522#comment-17039522
]
Jun Rao commented on KAFKA-6266:
[~david.mao]: I merged the PR to trunk. However, it doesn't apply to 2.4. Could
you submit a separate PR for the 2.4 branch? Thanks.
> Kafka 1.0.0 : Repeated occurrence of WARN Resetting first dirty offset of
> __consumer_offsets-xx to log start offset 203569 since the checkpointed
> offset 120955 is invalid. (kafka.log.LogCleanerManager$)
> --
>
> Key: KAFKA-6266
> URL: https://issues.apache.org/jira/browse/KAFKA-6266
> Project: Kafka
> Issue Type: Bug
> Components: offset manager
>Affects Versions: 1.0.0, 1.0.1
> Environment: CentOS 7, Apache kafka_2.12-1.0.0
>Reporter: VinayKumar
>Assignee: David Mao
>Priority: Major
> Fix For: 2.5.0, 2.4.1
>
>
> I upgraded Kafka from 0.10.2.1 to 1.0.0 version. From then, I see the below
> warnings in the log.
> I'm seeing these continuously in the log, and want these to be fixed- so
> that they wont repeat. Can someone please help me in fixing the below
> warnings.
> {code}
> WARN Resetting first dirty offset of __consumer_offsets-17 to log start
> offset 3346 since the checkpointed offset 3332 is invalid.
> (kafka.log.LogCleanerManager$)
> WARN Resetting first dirty offset of __consumer_offsets-23 to log start
> offset 4 since the checkpointed offset 1 is invalid.
> (kafka.log.LogCleanerManager$)
> WARN Resetting first dirty offset of __consumer_offsets-19 to log start
> offset 203569 since the checkpointed offset 120955 is invalid.
> (kafka.log.LogCleanerManager$)
> WARN Resetting first dirty offset of __consumer_offsets-35 to log start
> offset 16957 since the checkpointed offset 7 is invalid.
> (kafka.log.LogCleanerManager$)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-6266) Kafka 1.0.0 : Repeated occurrence of WARN Resetting first dirty offset of __consumer_offsets-xx to log start offset 203569 since the checkpointed offset 120955 is inval
[
https://issues.apache.org/jira/browse/KAFKA-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039518#comment-17039518
]
ASF GitHub Bot commented on KAFKA-6266:
---
junrao commented on pull request #8089: KAFKA-6266: Repeated occurrence of WARN
Resetting first dirty offset
URL: https://github.com/apache/kafka/pull/8089
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Kafka 1.0.0 : Repeated occurrence of WARN Resetting first dirty offset of
> __consumer_offsets-xx to log start offset 203569 since the checkpointed
> offset 120955 is invalid. (kafka.log.LogCleanerManager$)
> --
>
> Key: KAFKA-6266
> URL: https://issues.apache.org/jira/browse/KAFKA-6266
> Project: Kafka
> Issue Type: Bug
> Components: offset manager
>Affects Versions: 1.0.0, 1.0.1
> Environment: CentOS 7, Apache kafka_2.12-1.0.0
>Reporter: VinayKumar
>Assignee: David Mao
>Priority: Major
> Fix For: 2.5.0, 2.4.1
>
>
> I upgraded Kafka from 0.10.2.1 to 1.0.0 version. From then, I see the below
> warnings in the log.
> I'm seeing these continuously in the log, and want these to be fixed- so
> that they wont repeat. Can someone please help me in fixing the below
> warnings.
> {code}
> WARN Resetting first dirty offset of __consumer_offsets-17 to log start
> offset 3346 since the checkpointed offset 3332 is invalid.
> (kafka.log.LogCleanerManager$)
> WARN Resetting first dirty offset of __consumer_offsets-23 to log start
> offset 4 since the checkpointed offset 1 is invalid.
> (kafka.log.LogCleanerManager$)
> WARN Resetting first dirty offset of __consumer_offsets-19 to log start
> offset 203569 since the checkpointed offset 120955 is invalid.
> (kafka.log.LogCleanerManager$)
> WARN Resetting first dirty offset of __consumer_offsets-35 to log start
> offset 16957 since the checkpointed offset 7 is invalid.
> (kafka.log.LogCleanerManager$)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9568) Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership
[ https://issues.apache.org/jira/browse/KAFKA-9568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039505#comment-17039505 ] Boyang Chen commented on KAFKA-9568: Thanks for reporting this, I'm not sure I fully understood the question here. So let's suppose to have two instances A & B, we first migrate A to another host so that it's APPLICATION_SERVER_CONFIG would change, while instance B should still be using the same ip address and port as it is not restarting. So why would B get an invalid config in this case? > Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership > > > Key: KAFKA-9568 > URL: https://issues.apache.org/jira/browse/KAFKA-9568 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.0 >Reporter: David J. Garcia >Priority: Major > > A kstreams application with static membership, and > StreamsConfg.APPLICATION_SERVER_CONFIG set, will NOT update old server config > upon restart of application on new host. > Steps to reproduce: > > # start two kstreams applications (with same consumer group) and enable > static membership (and set application server config to :) > # kill one of the applications and restart it on a new host(with new ip) > before timeout ends (so that rebalancing doesn't occur). > # the other kstreams application will now have an invalid > application_server_config > Possible fix: > If an application restarts with a new host/identity..etc, it could trigger a > "light-rebalance" where the other applications in the consumer group don't > change partition assignments ,but instead just get their configuration > updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9566) ProcessorContextImpl#forward throws NullPointerException if invoked from DeserializationExceptionHandler
[
https://issues.apache.org/jira/browse/KAFKA-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039478#comment-17039478
]
Tomas Mi commented on KAFKA-9566:
-
Thank you Matthias :) Re this issue, if it is just the matter of JavaDocs and
changing to some nicer exception, I think its priority can be decreased.
> ProcessorContextImpl#forward throws NullPointerException if invoked from
> DeserializationExceptionHandler
>
>
> Key: KAFKA-9566
> URL: https://issues.apache.org/jira/browse/KAFKA-9566
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Tomas Mi
>Priority: Major
>
> Hi, I am trying to implement custom DeserializationExceptionHandler which
> would forward an exception to downstream processor(s), but
> ProcessorContextImpl#forward throws a NullPointerException if invoked from
> this custom handler.
> Handler implementation:
> {code:title=MyDeserializationExceptionHandler.java}
> public class MyDeserializationExceptionHandler implements
> DeserializationExceptionHandler {
> @Override
> public void configure(Map configs) {
> }
> @Override
> public DeserializationHandlerResponse handle(ProcessorContext context,
> ConsumerRecord record, Exception exception) {
> context.forward(null, exception, To.child("error-processor"));
> return DeserializationHandlerResponse.CONTINUE;
> }
> }
> {code}
> Handler is wired as default deserialization exception handler:
> {code}
> private TopologyTestDriver initializeTestDriver(StreamsBuilder
> streamBuilder) {
> Topology topology = streamBuilder.build();
> Properties props = new Properties();
> props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG,
> "my-test-application");
> props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
> "dummy:1234");
> props.setProperty(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> StreamsConfig.EXACTLY_ONCE);
>
> props.setProperty(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
> MyDeserializationExceptionHandler.class.getName());
> return new TopologyTestDriver(topology, props);
> }
> {code}
>
> Exception stacktrace:
> {noformat}
> org.apache.kafka.streams.errors.StreamsException: Fatal user code error in
> deserialization error callback
> at
> org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:76)
> at
> org.apache.kafka.streams.processor.internals.RecordQueue.maybeUpdateTimestamp(RecordQueue.java:160)
> at
> org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:101)
> at
> org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:136)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:742)
> at
> org.apache.kafka.streams.TopologyTestDriver.pipeInput(TopologyTestDriver.java:392)
> ...
> Caused by: java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:165)
> at
> MyDeserializationExceptionHandler.handle(NewExceptionHandlerTest.java:204)
> at
> org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:70)
> ... 33 more
> {noformat}
> Neither DeserializationExceptionHandler, nor ProcessorContext javadocs
> mention that ProcessorContext#forward(...) must not be invoked from
> DeserializationExceptionHandler, so I assume that this is a defect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9570) SSL cannot be configured for Connect in standalone mode
[
https://issues.apache.org/jira/browse/KAFKA-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039477#comment-17039477
]
ASF GitHub Bot commented on KAFKA-9570:
---
C0urante commented on pull request #8135: KAFKA-9570: Define SSL configs in all
worker config classes, not just distributed
URL: https://github.com/apache/kafka/pull/8135
[Jira](https://issues.apache.org/jira/browse/KAFKA-9570)
All SSL-related configs are currently defined only in the
`DistributedConfig` class, even though they are applicable for standalone mode
as well (since standalone mode also supports the Connect REST API). Because of
how these configs are parsed by the framework, it's currently impossible to
configure Connect in standalone mode to use SSL for the REST API with a
password-protected keystore, key, or truststore, and even if no password
protection is required, SSL configs will not be picked up correctly by the
worker if any of the worker configs start with the `listeners.https.` prefix.
These changes define the relevant SSL-related configs in the parent
`WorkerConfig` class, which should fix how they are picked up in standalone
mode.
A new unit test is added to verify that the `StandaloneConfig` picks up
these configs correctly.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> SSL cannot be configured for Connect in standalone mode
> ---
>
> Key: KAFKA-9570
> URL: https://issues.apache.org/jira/browse/KAFKA-9570
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.0.2, 2.3.0, 2.1.2,
> 2.2.1, 2.2.2, 2.4.0, 2.3.1, 2.2.3, 2.5.0, 2.3.2, 2.4.1
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
>
> When Connect is brought up in standalone, if the worker config contains _any_
> properties that begin with the {{listeners.https.}} prefix, SSL will not be
> enabled on the worker.
> This is because the relevant SSL configs are only defined in the [distributed
> worker
> config|https://github.com/apache/kafka/blob/ebcdcd9fa94efbff80e52b02c85d4a61c09f850b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedConfig.java#L260]
> instead of the [superclass worker
> config|https://github.com/apache/kafka/blob/ebcdcd9fa94efbff80e52b02c85d4a61c09f850b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerConfig.java].
> This, in conjunction with [a call
> to|https://github.com/apache/kafka/blob/ebcdcd9fa94efbff80e52b02c85d4a61c09f850b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/util/SSLUtils.java#L42]
>
> [AbstractConfig::valuesWithPrefixAllOrNothing|https://github.com/apache/kafka/blob/ebcdcd9fa94efbff80e52b02c85d4a61c09f850b/clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java],
> causes all configs not defined in the {{WorkerConfig}} used by the worker to
> be silently dropped when the worker configures its REST server if there is at
> least one config present with the {{listeners.https.}} prefix.
> Unfortunately, the workaround of specifying all SSL configs without the
> {{listeners.https.}} prefix will also fail if any passwords need to be
> specified. This is because the password values in the {{Map}} returned from
> {{AbstractConfig::valuesWithPrefixAllOrNothing}} aren't parsed as passwords,
> but the [framework expects them to
> be|https://github.com/apache/kafka/blob/ebcdcd9fa94efbff80e52b02c85d4a61c09f850b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/util/SSLUtils.java#L87].
> However, if no keystore, truststore, or key passwords need to be configured,
> then it should be possible to work around the issue by specifying all of
> those configurations without a prefix (as long as they don't conflict with
> any other configs in that namespace).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9570) SSL cannot be configured for Connect in standalone mode
Chris Egerton created KAFKA-9570:
Summary: SSL cannot be configured for Connect in standalone mode
Key: KAFKA-9570
URL: https://issues.apache.org/jira/browse/KAFKA-9570
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 2.3.1, 2.4.0, 2.2.2, 2.2.1, 2.3.0, 2.1.1, 2.2.0, 2.1.0,
2.0.1, 2.0.0, 2.0.2, 2.1.2, 2.2.3, 2.5.0, 2.3.2, 2.4.1
Reporter: Chris Egerton
Assignee: Chris Egerton
When Connect is brought up in standalone, if the worker config contains _any_
properties that begin with the {{listeners.https.}} prefix, SSL will not be
enabled on the worker.
This is because the relevant SSL configs are only defined in the [distributed
worker
config|https://github.com/apache/kafka/blob/ebcdcd9fa94efbff80e52b02c85d4a61c09f850b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedConfig.java#L260]
instead of the [superclass worker
config|https://github.com/apache/kafka/blob/ebcdcd9fa94efbff80e52b02c85d4a61c09f850b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerConfig.java].
This, in conjunction with [a call
to|https://github.com/apache/kafka/blob/ebcdcd9fa94efbff80e52b02c85d4a61c09f850b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/util/SSLUtils.java#L42]
[AbstractConfig::valuesWithPrefixAllOrNothing|https://github.com/apache/kafka/blob/ebcdcd9fa94efbff80e52b02c85d4a61c09f850b/clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java],
causes all configs not defined in the {{WorkerConfig}} used by the worker to
be silently dropped when the worker configures its REST server if there is at
least one config present with the {{listeners.https.}} prefix.
Unfortunately, the workaround of specifying all SSL configs without the
{{listeners.https.}} prefix will also fail if any passwords need to be
specified. This is because the password values in the {{Map}} returned from
{{AbstractConfig::valuesWithPrefixAllOrNothing}} aren't parsed as passwords,
but the [framework expects them to
be|https://github.com/apache/kafka/blob/ebcdcd9fa94efbff80e52b02c85d4a61c09f850b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/util/SSLUtils.java#L87].
However, if no keystore, truststore, or key passwords need to be configured,
then it should be possible to work around the issue by specifying all of those
configurations without a prefix (as long as they don't conflict with any other
configs in that namespace).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9566) ProcessorContextImpl#forward throws NullPointerException if invoked from DeserializationExceptionHandler
[
https://issues.apache.org/jira/browse/KAFKA-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039474#comment-17039474
]
Matthias J. Sax commented on KAFKA-9566:
In general it might be possible, but currently the assumption is that if a
deserialization error occurs, it's not possible to fix the issue and that the
message can only be dropped. Changing this would be a new feature and would
require a new Jira and a KIP to add this capabilities.
What you could do atm is, to use a custom deserializer that wraps the original
deserializer and catch any exception from the original deserializer and than
fix the issue and return the data you want to return.
> ProcessorContextImpl#forward throws NullPointerException if invoked from
> DeserializationExceptionHandler
>
>
> Key: KAFKA-9566
> URL: https://issues.apache.org/jira/browse/KAFKA-9566
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Tomas Mi
>Priority: Major
>
> Hi, I am trying to implement custom DeserializationExceptionHandler which
> would forward an exception to downstream processor(s), but
> ProcessorContextImpl#forward throws a NullPointerException if invoked from
> this custom handler.
> Handler implementation:
> {code:title=MyDeserializationExceptionHandler.java}
> public class MyDeserializationExceptionHandler implements
> DeserializationExceptionHandler {
> @Override
> public void configure(Map configs) {
> }
> @Override
> public DeserializationHandlerResponse handle(ProcessorContext context,
> ConsumerRecord record, Exception exception) {
> context.forward(null, exception, To.child("error-processor"));
> return DeserializationHandlerResponse.CONTINUE;
> }
> }
> {code}
> Handler is wired as default deserialization exception handler:
> {code}
> private TopologyTestDriver initializeTestDriver(StreamsBuilder
> streamBuilder) {
> Topology topology = streamBuilder.build();
> Properties props = new Properties();
> props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG,
> "my-test-application");
> props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
> "dummy:1234");
> props.setProperty(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> StreamsConfig.EXACTLY_ONCE);
>
> props.setProperty(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
> MyDeserializationExceptionHandler.class.getName());
> return new TopologyTestDriver(topology, props);
> }
> {code}
>
> Exception stacktrace:
> {noformat}
> org.apache.kafka.streams.errors.StreamsException: Fatal user code error in
> deserialization error callback
> at
> org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:76)
> at
> org.apache.kafka.streams.processor.internals.RecordQueue.maybeUpdateTimestamp(RecordQueue.java:160)
> at
> org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:101)
> at
> org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:136)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:742)
> at
> org.apache.kafka.streams.TopologyTestDriver.pipeInput(TopologyTestDriver.java:392)
> ...
> Caused by: java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:165)
> at
> MyDeserializationExceptionHandler.handle(NewExceptionHandlerTest.java:204)
> at
> org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:70)
> ... 33 more
> {noformat}
> Neither DeserializationExceptionHandler, nor ProcessorContext javadocs
> mention that ProcessorContext#forward(...) must not be invoked from
> DeserializationExceptionHandler, so I assume that this is a defect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-8858) Kafka Streams - Failed to Rebalance Error and stream consumer stuck for some reason
[
https://issues.apache.org/jira/browse/KAFKA-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039469#comment-17039469
]
Matthias J. Sax commented on KAFKA-8858:
Thanks [~ableegoldman]! Makes sense.
> Kafka Streams - Failed to Rebalance Error and stream consumer stuck for some
> reason
> ---
>
> Key: KAFKA-8858
> URL: https://issues.apache.org/jira/browse/KAFKA-8858
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.1
> Environment: Apache Kafka 2.1.1
>Reporter: Ante B.
>Priority: Major
> Labels: Stream, consumer, corrupt, offset, rebalance,
> transactions
>
> I have a basic Kafka Streams application that reads from a {{topic}},
> performs a rolling aggregate, and performs a join to publish to an
> {{agg_topic}}. Our project has the timeout failure in Kafka 2.1.1 env and we
> don't know the reason yet.
> Our stream consumer stuck for some reason.
> After we changed our group id to another one it became normal. So seems
> offset data for this consumer is corrupted.
> Can you help us please to resolve this problem to be able to revert us to the
> previous consumer name because we have many inconveniences due to this.
> Ping me pls if you will need some additional info.
> Our temporary workaround is to disable the {{exactly_once}} config which
> skips the initializing transactional state. Also offset reseted for corrupted
> partition, with no effect.
> Full problem description in log:
> {code:java}
> [2019-08-30 14:20:02.168] [abc-streamer-StreamThread-21] ERROR
> org.apache.kafka.streams.processor.internals.StreamThread:273 - stream-thread
> [abc-streamer-StreamThread-21] Error caught during partition assignment, will
> abort the current process and re-throw at the end of rebalance: {}
> org.apache.kafka.common.errors.TimeoutException: Timeout expired while
> initializing transactional state in 6ms.
> [2019-08-30 14:21:35.407] [abc-streamer-StreamThread-14] ERROR
> org.apache.kafka.streams.processor.internals.StreamThread:273 - stream-thread
> [abc-streamer-StreamThread-14] Error caught during partition assignment, will
> abort the current process and re-throw at the end of rebalance: {}
> org.apache.kafka.common.errors.TimeoutException: Timeout expired while
> initializing transactional state in 6ms.
> [2019-08-30 14:22:58.487] [abc-streamer-StreamThread-13] ERROR
> org.apache.kafka.streams.processor.internals.StreamThread:273 - stream-thread
> [abc-streamer-StreamThread-13] Error caught during partition assignment, will
> abort the current process and re-throw at the end of rebalance: {}
> org.apache.kafka.common.errors.TimeoutException: Timeout expired while
> initializing transactional state in 6ms.
> {noformat}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9546) Make FileStreamSourceTask extendable with generic streams
[ https://issues.apache.org/jira/browse/KAFKA-9546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Galyo updated KAFKA-9546: --- Labels: connect-api (was: ) > Make FileStreamSourceTask extendable with generic streams > - > > Key: KAFKA-9546 > URL: https://issues.apache.org/jira/browse/KAFKA-9546 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Csaba Galyo >Assignee: Csaba Galyo >Priority: Major > Labels: connect-api > Original Estimate: 4h > Remaining Estimate: 4h > > Use case: I want to read a ZIP compressed text file with a file connector and > send it to Kafka. > Currently, we have FileStreamSourceConnector which reads a \n delimited text > file. This connector always returns a task of type FileStreamSourceTask. > The FileStreamSourceTask reads from stdio or opens a file InputStream. The > issue with this approach is that the input needs to be a text file, otherwise > it won't work. > The code should be modified so that users could change the default > InputStream to eg. ZipInputStream, or any other format. The code is currently > written in such a way that it's not possible to extend it, we cannot use a > different input stream. > See example here where the code got copy-pasted just so it could read from a > ZstdInputStream (which reads ZSTD compressed files): > [https://github.com/gcsaba2/kafka-zstd/tree/master/src/main/java/org/apache/kafka/connect/file] > > I suggest 2 changes: > # FileStreamSourceConnector should be extendable to return tasks of > different types. These types would be input by the user through the > configuration map > # FileStreamSourceTask should be modified so it could be extended and child > classes could define different input streams. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9546) Make FileStreamSourceTask extendable with generic streams
[ https://issues.apache.org/jira/browse/KAFKA-9546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039452#comment-17039452 ] ASF GitHub Bot commented on KAFKA-9546: --- gcsaba2 commented on pull request #8134: KAFKA-9546 Allow custom tasks through configuration URL: https://github.com/apache/kafka/pull/8134 Currently FileStreamSourceConnector can only return a task of type FileStreamSourceTask. With this change the users can override it and provide a custom task class via configuration. Testing was done via unit tests. There's one positive case (custom Task class provided through config) and one negative (invalid class java.io.File was provided). The already existing unit tests are testing the default behavior, when FileStreamSourceTask is used. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make FileStreamSourceTask extendable with generic streams > - > > Key: KAFKA-9546 > URL: https://issues.apache.org/jira/browse/KAFKA-9546 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Csaba Galyo >Assignee: Csaba Galyo >Priority: Major > Original Estimate: 4h > Remaining Estimate: 4h > > Use case: I want to read a ZIP compressed text file with a file connector and > send it to Kafka. > Currently, we have FileStreamSourceConnector which reads a \n delimited text > file. This connector always returns a task of type FileStreamSourceTask. > The FileStreamSourceTask reads from stdio or opens a file InputStream. The > issue with this approach is that the input needs to be a text file, otherwise > it won't work. > The code should be modified so that users could change the default > InputStream to eg. ZipInputStream, or any other format. The code is currently > written in such a way that it's not possible to extend it, we cannot use a > different input stream. > See example here where the code got copy-pasted just so it could read from a > ZstdInputStream (which reads ZSTD compressed files): > [https://github.com/gcsaba2/kafka-zstd/tree/master/src/main/java/org/apache/kafka/connect/file] > > I suggest 2 changes: > # FileStreamSourceConnector should be extendable to return tasks of > different types. These types would be input by the user through the > configuration map > # FileStreamSourceTask should be modified so it could be extended and child > classes could define different input streams. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9566) ProcessorContextImpl#forward throws NullPointerException if invoked from DeserializationExceptionHandler
[
https://issues.apache.org/jira/browse/KAFKA-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039444#comment-17039444
]
Tomas Mi commented on KAFKA-9566:
-
Wouldn't it be possible to support such use case instead of throwing
UnsupportedOperationException? Per my understanding the issue relates to
missing child processors (or only partly initialized ProcessorContext), but
durring message deserialization phaze I think child processors should be known,
since that should happen in source processor. I am trying to implement
exception handling while maintaining exactly once delivery guaratee and it
looks like that this would be an ideal way. Otherwise do you know if there is
another way to achieve the same?
> ProcessorContextImpl#forward throws NullPointerException if invoked from
> DeserializationExceptionHandler
>
>
> Key: KAFKA-9566
> URL: https://issues.apache.org/jira/browse/KAFKA-9566
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Tomas Mi
>Priority: Major
>
> Hi, I am trying to implement custom DeserializationExceptionHandler which
> would forward an exception to downstream processor(s), but
> ProcessorContextImpl#forward throws a NullPointerException if invoked from
> this custom handler.
> Handler implementation:
> {code:title=MyDeserializationExceptionHandler.java}
> public class MyDeserializationExceptionHandler implements
> DeserializationExceptionHandler {
> @Override
> public void configure(Map configs) {
> }
> @Override
> public DeserializationHandlerResponse handle(ProcessorContext context,
> ConsumerRecord record, Exception exception) {
> context.forward(null, exception, To.child("error-processor"));
> return DeserializationHandlerResponse.CONTINUE;
> }
> }
> {code}
> Handler is wired as default deserialization exception handler:
> {code}
> private TopologyTestDriver initializeTestDriver(StreamsBuilder
> streamBuilder) {
> Topology topology = streamBuilder.build();
> Properties props = new Properties();
> props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG,
> "my-test-application");
> props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
> "dummy:1234");
> props.setProperty(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> StreamsConfig.EXACTLY_ONCE);
>
> props.setProperty(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
> MyDeserializationExceptionHandler.class.getName());
> return new TopologyTestDriver(topology, props);
> }
> {code}
>
> Exception stacktrace:
> {noformat}
> org.apache.kafka.streams.errors.StreamsException: Fatal user code error in
> deserialization error callback
> at
> org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:76)
> at
> org.apache.kafka.streams.processor.internals.RecordQueue.maybeUpdateTimestamp(RecordQueue.java:160)
> at
> org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:101)
> at
> org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:136)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:742)
> at
> org.apache.kafka.streams.TopologyTestDriver.pipeInput(TopologyTestDriver.java:392)
> ...
> Caused by: java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:165)
> at
> MyDeserializationExceptionHandler.handle(NewExceptionHandlerTest.java:204)
> at
> org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:70)
> ... 33 more
> {noformat}
> Neither DeserializationExceptionHandler, nor ProcessorContext javadocs
> mention that ProcessorContext#forward(...) must not be invoked from
> DeserializationExceptionHandler, so I assume that this is a defect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-8025) Flaky Test RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
[
https://issues.apache.org/jira/browse/KAFKA-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax resolved KAFKA-8025.
Resolution: Fixed
> Flaky Test
> RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
>
>
> Key: KAFKA-8025
> URL: https://issues.apache.org/jira/browse/KAFKA-8025
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.2.0
>Reporter: Konstantine Karantasis
>Assignee: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.2.3, 2.5.0, 2.3.2, 2.4.1
>
>
> At least one occurence where the following unit test case failed on a jenkins
> job that didn't involve any related changes.
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/2783/consoleFull]
> I have not been able to reproduce it locally on Linux. (For instance 20
> consecutive runs of this class pass all test cases)
> {code:java}
> 14:06:13
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllDbOptionsCalls STARTED 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/streams/build/reports/testOutput/org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls.test.stdout
> 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllDbOptionsCalls FAILED 14:06:14
> java.lang.AssertionError: 14:06:14 Expected: a string matching the
> pattern 'Unexpected method call DBOptions\.baseBackgroundCompactions((.*
> 14:06:14 *)*):' 14:06:14 but: was "Unexpected method call
> DBOptions.baseBackgroundCompactions():\n DBOptions.close(): expected: 3,
> actual: 0" 14:06:14 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) 14:06:14
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) 14:06:14
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.verifyDBOptionsMethodCall(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:121)
> 14:06:14 at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:101)
> 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllColumnFamilyCalls STARTED 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllColumnFamilyCalls PASSED
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8025) Flaky Test RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
[
https://issues.apache.org/jira/browse/KAFKA-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-8025:
---
Fix Version/s: 2.4.1
2.3.2
2.2.3
> Flaky Test
> RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
>
>
> Key: KAFKA-8025
> URL: https://issues.apache.org/jira/browse/KAFKA-8025
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.2.0
>Reporter: Konstantine Karantasis
>Assignee: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.2.3, 2.5.0, 2.3.2, 2.4.1
>
>
> At least one occurence where the following unit test case failed on a jenkins
> job that didn't involve any related changes.
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/2783/consoleFull]
> I have not been able to reproduce it locally on Linux. (For instance 20
> consecutive runs of this class pass all test cases)
> {code:java}
> 14:06:13
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllDbOptionsCalls STARTED 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/streams/build/reports/testOutput/org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls.test.stdout
> 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllDbOptionsCalls FAILED 14:06:14
> java.lang.AssertionError: 14:06:14 Expected: a string matching the
> pattern 'Unexpected method call DBOptions\.baseBackgroundCompactions((.*
> 14:06:14 *)*):' 14:06:14 but: was "Unexpected method call
> DBOptions.baseBackgroundCompactions():\n DBOptions.close(): expected: 3,
> actual: 0" 14:06:14 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) 14:06:14
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) 14:06:14
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.verifyDBOptionsMethodCall(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:121)
> 14:06:14 at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:101)
> 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllColumnFamilyCalls STARTED 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllColumnFamilyCalls PASSED
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-8025) Flaky Test RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
[
https://issues.apache.org/jira/browse/KAFKA-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-8025:
---
Affects Version/s: (was: 2.3.0)
2.2.0
> Flaky Test
> RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
>
>
> Key: KAFKA-8025
> URL: https://issues.apache.org/jira/browse/KAFKA-8025
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.2.0
>Reporter: Konstantine Karantasis
>Assignee: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.5.0
>
>
> At least one occurence where the following unit test case failed on a jenkins
> job that didn't involve any related changes.
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/2783/consoleFull]
> I have not been able to reproduce it locally on Linux. (For instance 20
> consecutive runs of this class pass all test cases)
> {code:java}
> 14:06:13
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllDbOptionsCalls STARTED 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/streams/build/reports/testOutput/org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls.test.stdout
> 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllDbOptionsCalls FAILED 14:06:14
> java.lang.AssertionError: 14:06:14 Expected: a string matching the
> pattern 'Unexpected method call DBOptions\.baseBackgroundCompactions((.*
> 14:06:14 *)*):' 14:06:14 but: was "Unexpected method call
> DBOptions.baseBackgroundCompactions():\n DBOptions.close(): expected: 3,
> actual: 0" 14:06:14 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) 14:06:14
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) 14:06:14
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.verifyDBOptionsMethodCall(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:121)
> 14:06:14 at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:101)
> 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllColumnFamilyCalls STARTED 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllColumnFamilyCalls PASSED
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-9546) Make FileStreamSourceTask extendable with generic streams
[ https://issues.apache.org/jira/browse/KAFKA-9546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang reassigned KAFKA-9546: Assignee: Csaba Galyo > Make FileStreamSourceTask extendable with generic streams > - > > Key: KAFKA-9546 > URL: https://issues.apache.org/jira/browse/KAFKA-9546 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Csaba Galyo >Assignee: Csaba Galyo >Priority: Major > Original Estimate: 4h > Remaining Estimate: 4h > > Use case: I want to read a ZIP compressed text file with a file connector and > send it to Kafka. > Currently, we have FileStreamSourceConnector which reads a \n delimited text > file. This connector always returns a task of type FileStreamSourceTask. > The FileStreamSourceTask reads from stdio or opens a file InputStream. The > issue with this approach is that the input needs to be a text file, otherwise > it won't work. > The code should be modified so that users could change the default > InputStream to eg. ZipInputStream, or any other format. The code is currently > written in such a way that it's not possible to extend it, we cannot use a > different input stream. > See example here where the code got copy-pasted just so it could read from a > ZstdInputStream (which reads ZSTD compressed files): > [https://github.com/gcsaba2/kafka-zstd/tree/master/src/main/java/org/apache/kafka/connect/file] > > I suggest 2 changes: > # FileStreamSourceConnector should be extendable to return tasks of > different types. These types would be input by the user through the > configuration map > # FileStreamSourceTask should be modified so it could be extended and child > classes could define different input streams. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9430) Tighten up lag estimates when source topic optimization is on
[ https://issues.apache.org/jira/browse/KAFKA-9430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apurva Mehta updated KAFKA-9430: Priority: Major (was: Blocker) > Tighten up lag estimates when source topic optimization is on > -- > > Key: KAFKA-9430 > URL: https://issues.apache.org/jira/browse/KAFKA-9430 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.5.0 >Reporter: Vinoth Chandar >Assignee: Vinoth Chandar >Priority: Major > > Right now, we use _endOffsets_ of the source topic for the computation. For > "optimized" changelogs, this will be wrong, strictly speaking, but it's an > over-estimate (which seems better than an under-estimate), and it's also > still an apples-to-apples comparison, since all replicas would use the same > upper bound to compute their lags, so the "pick the freshest" replica is > still going to pick the right one. > The current implementation is technically correct, within the documented > behavior that the result is an "estimate", but I marked it as a blocker to be > sure that we revisit it after ongoing work to refactor the task management in > Streams is complete. If it becomes straightforward to tighten up the > estimate, we should go ahead and do it. Otherwise, we can downgrade the > priority of the ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8025) Flaky Test RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
[
https://issues.apache.org/jira/browse/KAFKA-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039392#comment-17039392
]
ASF GitHub Bot commented on KAFKA-8025:
---
mjsax commented on pull request #8126: KAFKA-8025: Fix flaky RocksDB test
URL: https://github.com/apache/kafka/pull/8126
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flaky Test
> RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
>
>
> Key: KAFKA-8025
> URL: https://issues.apache.org/jira/browse/KAFKA-8025
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.3.0
>Reporter: Konstantine Karantasis
>Assignee: Bill Bejeck
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.5.0
>
>
> At least one occurence where the following unit test case failed on a jenkins
> job that didn't involve any related changes.
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/2783/consoleFull]
> I have not been able to reproduce it locally on Linux. (For instance 20
> consecutive runs of this class pass all test cases)
> {code:java}
> 14:06:13
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllDbOptionsCalls STARTED 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/streams/build/reports/testOutput/org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls.test.stdout
> 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllDbOptionsCalls FAILED 14:06:14
> java.lang.AssertionError: 14:06:14 Expected: a string matching the
> pattern 'Unexpected method call DBOptions\.baseBackgroundCompactions((.*
> 14:06:14 *)*):' 14:06:14 but: was "Unexpected method call
> DBOptions.baseBackgroundCompactions():\n DBOptions.close(): expected: 3,
> actual: 0" 14:06:14 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) 14:06:14
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) 14:06:14
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.verifyDBOptionsMethodCall(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:121)
> 14:06:14 at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:101)
> 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllColumnFamilyCalls STARTED 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllColumnFamilyCalls PASSED
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-8025) Flaky Test RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
[
https://issues.apache.org/jira/browse/KAFKA-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reassigned KAFKA-8025:
--
Assignee: Matthias J. Sax (was: Bill Bejeck)
> Flaky Test
> RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls
>
>
> Key: KAFKA-8025
> URL: https://issues.apache.org/jira/browse/KAFKA-8025
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.3.0
>Reporter: Konstantine Karantasis
>Assignee: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.5.0
>
>
> At least one occurence where the following unit test case failed on a jenkins
> job that didn't involve any related changes.
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/2783/consoleFull]
> I have not been able to reproduce it locally on Linux. (For instance 20
> consecutive runs of this class pass all test cases)
> {code:java}
> 14:06:13
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllDbOptionsCalls STARTED 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/streams/build/reports/testOutput/org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls.test.stdout
> 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllDbOptionsCalls FAILED 14:06:14
> java.lang.AssertionError: 14:06:14 Expected: a string matching the
> pattern 'Unexpected method call DBOptions\.baseBackgroundCompactions((.*
> 14:06:14 *)*):' 14:06:14 but: was "Unexpected method call
> DBOptions.baseBackgroundCompactions():\n DBOptions.close(): expected: 3,
> actual: 0" 14:06:14 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) 14:06:14
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) 14:06:14
> at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.verifyDBOptionsMethodCall(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:121)
> 14:06:14 at
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls(RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java:101)
> 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllColumnFamilyCalls STARTED 14:06:14 14:06:14
> org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest
> > shouldForwardAllColumnFamilyCalls PASSED
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9541) Flaky Test DescribeConsumerGroupTest#testDescribeGroupMembersWithShortInitializationTimeout
[ https://issues.apache.org/jira/browse/KAFKA-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039391#comment-17039391 ] Matthias J. Sax commented on KAFKA-9541: [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/736/testReport/junit/kafka.admin/DescribeConsumerGroupTest/testDescribeGroupMembersWithShortInitializationTimeout/] DisconnectException again instead of Timeout. > Flaky Test > DescribeConsumerGroupTest#testDescribeGroupMembersWithShortInitializationTimeout > --- > > Key: KAFKA-9541 > URL: https://issues.apache.org/jira/browse/KAFKA-9541 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.4.0 >Reporter: huxihx >Assignee: huxihx >Priority: Major > > h3. Error Message > java.lang.AssertionError: assertion failed > h3. Stacktrace > java.lang.AssertionError: assertion failed at > scala.Predef$.assert(Predef.scala:267) at > kafka.admin.DescribeConsumerGroupTest.testDescribeGroupMembersWithShortInitializationTimeout(DescribeConsumerGroupTest.scala:630) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at > org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at > org.junit.runners.ParentRunner.run(ParentRunner.java:413) at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) > at > org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) > at jdk.internal.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) > at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at > org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) > at jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at >
[jira] [Commented] (KAFKA-9530) Flaky Test kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout
[
https://issues.apache.org/jira/browse/KAFKA-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039390#comment-17039390
]
Matthias J. Sax commented on KAFKA-9530:
[https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/736/testReport/junit/kafka.admin/DescribeConsumerGroupTest/testDescribeGroupWithShortInitializationTimeout/]
{quote}java.lang.AssertionError: expected: but was: at
org.junit.Assert.fail(Assert.java:89) at
org.junit.Assert.failNotEquals(Assert.java:835) at
org.junit.Assert.assertEquals(Assert.java:120) at
org.junit.Assert.assertEquals(Assert.java:146) at
kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout(DescribeConsumerGroupTest.scala:585){quote}
> Flaky Test
> kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout
>
>
> Key: KAFKA-9530
> URL: https://issues.apache.org/jira/browse/KAFKA-9530
> Project: Kafka
> Issue Type: Test
> Components: core
>Reporter: Bill Bejeck
>Priority: Major
> Labels: flaky-test, test
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/4570/testReport/junit/kafka.admin/DescribeConsumerGroupTest/testDescribeGroupWithShortInitializationTimeout/]
>
> {noformat}
> Error Messagejava.lang.AssertionError: assertion
> failedStacktracejava.lang.AssertionError: assertion failed
> at scala.Predef$.assert(Predef.scala:267)
> at
> kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout(DescribeConsumerGroupTest.scala:585)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at jdk.internal.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>
[jira] [Commented] (KAFKA-8858) Kafka Streams - Failed to Rebalance Error and stream consumer stuck for some reason
[
https://issues.apache.org/jira/browse/KAFKA-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039352#comment-17039352
]
Sophie Blee-Goldman commented on KAFKA-8858:
[~mjsax] I'm not sure – they're both timeouts that resolve to the same original
API but they timeout at different places since that API (initTransaction)
itself results in two calls. If it were the same root cause it would be weird
that in some cases it never made it past the first call while in others it
always succeeded on the first call, then timed out on the second. Not
impossible, but definitely not certain to be the same
> Kafka Streams - Failed to Rebalance Error and stream consumer stuck for some
> reason
> ---
>
> Key: KAFKA-8858
> URL: https://issues.apache.org/jira/browse/KAFKA-8858
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.1
> Environment: Apache Kafka 2.1.1
>Reporter: Ante B.
>Priority: Major
> Labels: Stream, consumer, corrupt, offset, rebalance,
> transactions
>
> I have a basic Kafka Streams application that reads from a {{topic}},
> performs a rolling aggregate, and performs a join to publish to an
> {{agg_topic}}. Our project has the timeout failure in Kafka 2.1.1 env and we
> don't know the reason yet.
> Our stream consumer stuck for some reason.
> After we changed our group id to another one it became normal. So seems
> offset data for this consumer is corrupted.
> Can you help us please to resolve this problem to be able to revert us to the
> previous consumer name because we have many inconveniences due to this.
> Ping me pls if you will need some additional info.
> Our temporary workaround is to disable the {{exactly_once}} config which
> skips the initializing transactional state. Also offset reseted for corrupted
> partition, with no effect.
> Full problem description in log:
> {code:java}
> [2019-08-30 14:20:02.168] [abc-streamer-StreamThread-21] ERROR
> org.apache.kafka.streams.processor.internals.StreamThread:273 - stream-thread
> [abc-streamer-StreamThread-21] Error caught during partition assignment, will
> abort the current process and re-throw at the end of rebalance: {}
> org.apache.kafka.common.errors.TimeoutException: Timeout expired while
> initializing transactional state in 6ms.
> [2019-08-30 14:21:35.407] [abc-streamer-StreamThread-14] ERROR
> org.apache.kafka.streams.processor.internals.StreamThread:273 - stream-thread
> [abc-streamer-StreamThread-14] Error caught during partition assignment, will
> abort the current process and re-throw at the end of rebalance: {}
> org.apache.kafka.common.errors.TimeoutException: Timeout expired while
> initializing transactional state in 6ms.
> [2019-08-30 14:22:58.487] [abc-streamer-StreamThread-13] ERROR
> org.apache.kafka.streams.processor.internals.StreamThread:273 - stream-thread
> [abc-streamer-StreamThread-13] Error caught during partition assignment, will
> abort the current process and re-throw at the end of rebalance: {}
> org.apache.kafka.common.errors.TimeoutException: Timeout expired while
> initializing transactional state in 6ms.
> {noformat}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9543) Consumer offset reset after new segment rolling
[
https://issues.apache.org/jira/browse/KAFKA-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039336#comment-17039336
]
Rafał Boniecki edited comment on KAFKA-9543 at 2/18/20 6:33 PM:
Yes it is consumer lag (every color represents different partition). As you can
see before update there was practically none of it (it can be easily deduced
when upgrade took place ;)). I will try to reproduce it and let you know how it
went.
was (Author: boniek):
Yes it is consumer lag. As you can see before update there was practically none
of it (it can be easily deduced when upgrade took place ;)). I will try to
reproduce it and let you know how it went.
> Consumer offset reset after new segment rolling
> ---
>
> Key: KAFKA-9543
> URL: https://issues.apache.org/jira/browse/KAFKA-9543
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Rafał Boniecki
>Priority: Major
> Attachments: Untitled.png
>
>
> After upgrade from kafka 2.1.1 to 2.4.0, I'm experiencing unexpected consumer
> offset resets.
> Consumer:
> {code:java}
> 2020-02-12T11:12:58.402+01:00 hostname 4a2a39a35a02
> [2020-02-12T11:12:58,402][INFO
> ][org.apache.kafka.clients.consumer.internals.Fetcher] [Consumer
> clientId=logstash-1, groupId=logstash] Fetch offset 1632750575 is out of
> range for partition stats-5, resetting offset
> {code}
> Broker:
> {code:java}
> 2020-02-12 11:12:58:400 CET INFO
> [data-plane-kafka-request-handler-1][kafka.log.Log] [Log partition=stats-5,
> dir=/kafka4/data] Rolled new log segment at offset 1632750565 in 2 ms.{code}
> All resets are perfectly correlated to rolling new segments at the broker -
> segment is rolled first, then, couple of ms later, reset on the consumer
> occurs. Attached is grafana graph with consumer lag per partition. All sudden
> spikes in lag are offset resets due to this bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9543) Consumer offset reset after new segment rolling
[
https://issues.apache.org/jira/browse/KAFKA-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039336#comment-17039336
]
Rafał Boniecki commented on KAFKA-9543:
---
Yes it is consumer lag. As you can see before update there was practically none
of it (it can be easily deduced when upgrade took place ;)). I will try to
reproduce it and let you know how it went.
> Consumer offset reset after new segment rolling
> ---
>
> Key: KAFKA-9543
> URL: https://issues.apache.org/jira/browse/KAFKA-9543
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Rafał Boniecki
>Priority: Major
> Attachments: Untitled.png
>
>
> After upgrade from kafka 2.1.1 to 2.4.0, I'm experiencing unexpected consumer
> offset resets.
> Consumer:
> {code:java}
> 2020-02-12T11:12:58.402+01:00 hostname 4a2a39a35a02
> [2020-02-12T11:12:58,402][INFO
> ][org.apache.kafka.clients.consumer.internals.Fetcher] [Consumer
> clientId=logstash-1, groupId=logstash] Fetch offset 1632750575 is out of
> range for partition stats-5, resetting offset
> {code}
> Broker:
> {code:java}
> 2020-02-12 11:12:58:400 CET INFO
> [data-plane-kafka-request-handler-1][kafka.log.Log] [Log partition=stats-5,
> dir=/kafka4/data] Rolled new log segment at offset 1632750565 in 2 ms.{code}
> All resets are perfectly correlated to rolling new segments at the broker -
> segment is rolled first, then, couple of ms later, reset on the consumer
> occurs. Attached is grafana graph with consumer lag per partition. All sudden
> spikes in lag are offset resets due to this bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9376) Plugin class loader not found using MM2
[
https://issues.apache.org/jira/browse/KAFKA-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039327#comment-17039327
]
Ryanne Dolan commented on KAFKA-9376:
-
Is it possible you all have the MM2 Connectors in your Connect plugin.path? I
wonder if you hand-copied the jars at one point, and then Connect is trying to
load the Connectors from both locations? I suspect that would cause this sort
of behavior.
> Plugin class loader not found using MM2
> ---
>
> Key: KAFKA-9376
> URL: https://issues.apache.org/jira/browse/KAFKA-9376
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.4.0
>Reporter: Sinóros-Szabó Péter
>Priority: Minor
>
> I am using MM2 (release 2.4.0 with scala 2.12) I geta bunch of classloader
> errors. MM2 seems to be working, but I do not know if all of it components
> are working as expected as this is the first time I use MM2.
> I run MM2 with the following command:
> {code:java}
> ./bin/connect-mirror-maker.sh config/connect-mirror-maker.properties
> {code}
> Errors are:
> {code:java}
> [2020-01-07 15:06:17,892] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,889] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,904] INFO ConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
> (org.apache.kafka.connect.runtime.ConnectorConfig:347)
> [2020-01-07 15:06:17,904] INFO EnrichedConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
>
> (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
> [2020-01-07 15:06:17,905] INFO TaskConfig values:
> task.class = class org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.TaskConfig:347)
> [2020-01-07 15:06:17,905] INFO Instantiated task MirrorHeartbeatConnector-0
> with version 1 of type org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.Worker:434){code}
> After a while, these errors are not logged any more.
> Config is:
> {code:java}
> clusters = eucmain, euwbackup
> eucmain.bootstrap.servers = kafka1:9092,kafka2:9092
> euwbackup.bootstrap.servers = 172.30.197.203:9092,172.30.213.104:9092
> eucmain->euwbackup.enabled = true
> eucmain->euwbackup.topics = .*
> eucmain->euwbackup.topics.blacklist = ^(kafka|kmf|__|pricing).*
> eucmain->euwbackup.rename.topics = false
> rename.topics = false
> eucmain->euwbackup.sync.topic.acls.enabled = false
> sync.topic.acls.enabled = false{code}
> Using OpenJDK 8 or 11, I get the same error.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9540) Application getting "Could not find the standby task 0_4 while closing it" error
[
https://issues.apache.org/jira/browse/KAFKA-9540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sophie Blee-Goldman resolved KAFKA-9540.
Fix Version/s: 2.6.0
Resolution: Fixed
> Application getting "Could not find the standby task 0_4 while closing it"
> error
>
>
> Key: KAFKA-9540
> URL: https://issues.apache.org/jira/browse/KAFKA-9540
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Badai Aqrandista
>Priority: Minor
> Fix For: 2.4.1, 2.6.0
>
>
> Because of this the following line, there is a possibility that some standby
> tasks might not be created:
> https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L436
> Then causing this line to not adding the task to standby task list:
> https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L299
> But this line assumes that all standby tasks are to be created and add it to
> the standby list:
> https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java#L168
> This results in user getting this error message on the next
> PARTITION_ASSIGNMENT state:
> {noformat}
> Could not find the standby task 0_4 while closing it
> (org.apache.kafka.streams.processor.internals.AssignedStandbyTasks:74)
> {noformat}
> But the harm caused by this issue is minimal: No standby task for some
> partitions. And it is recreated on the next rebalance anyway. So, I suggest
> lowering this message to WARN. Or probably check to WARN when standby task
> could not be created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9512) Flaky Test LagFetchIntegrationTest.shouldFetchLagsDuringRestoration
[
https://issues.apache.org/jira/browse/KAFKA-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar resolved KAFKA-9512.
---
Resolution: Fixed
Closing since the PR is now landed
> Flaky Test LagFetchIntegrationTest.shouldFetchLagsDuringRestoration
> ---
>
> Key: KAFKA-9512
> URL: https://issues.apache.org/jira/browse/KAFKA-9512
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.5.0
>Reporter: Matthias J. Sax
>Assignee: Vinoth Chandar
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.5.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/497/testReport/junit/org.apache.kafka.streams.integration/LagFetchIntegrationTest/shouldFetchLagsDuringRestoration/]
> {quote}java.lang.NullPointerException at
> org.apache.kafka.streams.integration.LagFetchIntegrationTest.shouldFetchLagsDuringRestoration(LagFetchIntegrationTest.java:306){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9566) ProcessorContextImpl#forward throws NullPointerException if invoked from DeserializationExceptionHandler
[
https://issues.apache.org/jira/browse/KAFKA-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039283#comment-17039283
]
Matthias J. Sax commented on KAFKA-9566:
Thanks for reporting this issue. We should update the JavaDocs accordingly. The
ides to pass in the context is to provide context information like topic name,
partitions, offset, timestamp of the record etc.
We should also throw a better exception, like "UnsupportedOperationException".
> ProcessorContextImpl#forward throws NullPointerException if invoked from
> DeserializationExceptionHandler
>
>
> Key: KAFKA-9566
> URL: https://issues.apache.org/jira/browse/KAFKA-9566
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.2.0
>Reporter: Tomas Mi
>Priority: Major
>
> Hi, I am trying to implement custom DeserializationExceptionHandler which
> would forward an exception to downstream processor(s), but
> ProcessorContextImpl#forward throws a NullPointerException if invoked from
> this custom handler.
> Handler implementation:
> {code:title=MyDeserializationExceptionHandler.java}
> public class MyDeserializationExceptionHandler implements
> DeserializationExceptionHandler {
> @Override
> public void configure(Map configs) {
> }
> @Override
> public DeserializationHandlerResponse handle(ProcessorContext context,
> ConsumerRecord record, Exception exception) {
> context.forward(null, exception, To.child("error-processor"));
> return DeserializationHandlerResponse.CONTINUE;
> }
> }
> {code}
> Handler is wired as default deserialization exception handler:
> {code}
> private TopologyTestDriver initializeTestDriver(StreamsBuilder
> streamBuilder) {
> Topology topology = streamBuilder.build();
> Properties props = new Properties();
> props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG,
> "my-test-application");
> props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
> "dummy:1234");
> props.setProperty(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> StreamsConfig.EXACTLY_ONCE);
>
> props.setProperty(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
> MyDeserializationExceptionHandler.class.getName());
> return new TopologyTestDriver(topology, props);
> }
> {code}
>
> Exception stacktrace:
> {noformat}
> org.apache.kafka.streams.errors.StreamsException: Fatal user code error in
> deserialization error callback
> at
> org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:76)
> at
> org.apache.kafka.streams.processor.internals.RecordQueue.maybeUpdateTimestamp(RecordQueue.java:160)
> at
> org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:101)
> at
> org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:136)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:742)
> at
> org.apache.kafka.streams.TopologyTestDriver.pipeInput(TopologyTestDriver.java:392)
> ...
> Caused by: java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:165)
> at
> MyDeserializationExceptionHandler.handle(NewExceptionHandlerTest.java:204)
> at
> org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:70)
> ... 33 more
> {noformat}
> Neither DeserializationExceptionHandler, nor ProcessorContext javadocs
> mention that ProcessorContext#forward(...) must not be invoked from
> DeserializationExceptionHandler, so I assume that this is a defect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9568) Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership
[ https://issues.apache.org/jira/browse/KAFKA-9568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-9568: --- Affects Version/s: (was: 2.4.0) 2.3.0 > Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership > > > Key: KAFKA-9568 > URL: https://issues.apache.org/jira/browse/KAFKA-9568 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.3.0 >Reporter: David J. Garcia >Priority: Major > > A kstreams application with static membership, and > StreamsConfg.APPLICATION_SERVER_CONFIG set, will NOT update old server config > upon restart of application on new host. > Steps to reproduce: > > # start two kstreams applications (with same consumer group) and enable > static membership (and set application server config to :) > # kill one of the applications and restart it on a new host(with new ip) > before timeout ends (so that rebalancing doesn't occur). > # the other kstreams application will now have an invalid > application_server_config > Possible fix: > If an application restarts with a new host/identity..etc, it could trigger a > "light-rebalance" where the other applications in the consumer group don't > change partition assignments ,but instead just get their configuration > updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9566) ProcessorContextImpl#forward throws NullPointerException if invoked from DeserializationExceptionHandler
[
https://issues.apache.org/jira/browse/KAFKA-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-9566:
---
Affects Version/s: (was: 2.2.0)
1.0.0
> ProcessorContextImpl#forward throws NullPointerException if invoked from
> DeserializationExceptionHandler
>
>
> Key: KAFKA-9566
> URL: https://issues.apache.org/jira/browse/KAFKA-9566
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.0.0
>Reporter: Tomas Mi
>Priority: Major
>
> Hi, I am trying to implement custom DeserializationExceptionHandler which
> would forward an exception to downstream processor(s), but
> ProcessorContextImpl#forward throws a NullPointerException if invoked from
> this custom handler.
> Handler implementation:
> {code:title=MyDeserializationExceptionHandler.java}
> public class MyDeserializationExceptionHandler implements
> DeserializationExceptionHandler {
> @Override
> public void configure(Map configs) {
> }
> @Override
> public DeserializationHandlerResponse handle(ProcessorContext context,
> ConsumerRecord record, Exception exception) {
> context.forward(null, exception, To.child("error-processor"));
> return DeserializationHandlerResponse.CONTINUE;
> }
> }
> {code}
> Handler is wired as default deserialization exception handler:
> {code}
> private TopologyTestDriver initializeTestDriver(StreamsBuilder
> streamBuilder) {
> Topology topology = streamBuilder.build();
> Properties props = new Properties();
> props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG,
> "my-test-application");
> props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
> "dummy:1234");
> props.setProperty(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> StreamsConfig.EXACTLY_ONCE);
>
> props.setProperty(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
> MyDeserializationExceptionHandler.class.getName());
> return new TopologyTestDriver(topology, props);
> }
> {code}
>
> Exception stacktrace:
> {noformat}
> org.apache.kafka.streams.errors.StreamsException: Fatal user code error in
> deserialization error callback
> at
> org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:76)
> at
> org.apache.kafka.streams.processor.internals.RecordQueue.maybeUpdateTimestamp(RecordQueue.java:160)
> at
> org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:101)
> at
> org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:136)
> at
> org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:742)
> at
> org.apache.kafka.streams.TopologyTestDriver.pipeInput(TopologyTestDriver.java:392)
> ...
> Caused by: java.lang.NullPointerException
> at
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:165)
> at
> MyDeserializationExceptionHandler.handle(NewExceptionHandlerTest.java:204)
> at
> org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:70)
> ... 33 more
> {noformat}
> Neither DeserializationExceptionHandler, nor ProcessorContext javadocs
> mention that ProcessorContext#forward(...) must not be invoked from
> DeserializationExceptionHandler, so I assume that this is a defect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9569) RemoteStorageManager implementation for HDFS storage.
[ https://issues.apache.org/jira/browse/KAFKA-9569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Duggana updated KAFKA-9569: -- Summary: RemoteStorageManager implementation for HDFS storage. (was: RSM implementation for HDFS storage.) > RemoteStorageManager implementation for HDFS storage. > - > > Key: KAFKA-9569 > URL: https://issues.apache.org/jira/browse/KAFKA-9569 > Project: Kafka > Issue Type: Sub-task > Components: core >Reporter: Satish Duggana >Assignee: Ying Zheng >Priority: Major > > This is about implementing `RemoteStorageManager` for HDFS to verify the > proposed SPIs are sufficient. It looks like the existing RSM interface should > be sufficient. If needed, we will discuss any required changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9569) RSM implementation for HDFS storage.
Satish Duggana created KAFKA-9569: - Summary: RSM implementation for HDFS storage. Key: KAFKA-9569 URL: https://issues.apache.org/jira/browse/KAFKA-9569 Project: Kafka Issue Type: Sub-task Components: core Reporter: Satish Duggana Assignee: Ying Zheng This is about implementing `RemoteStorageManager` for HDFS to verify the proposed SPIs are sufficient. It looks like the existing RSM interface should be sufficient. If needed, we will discuss any required changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9568) Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership
David J. Garcia created KAFKA-9568: -- Summary: Kstreams APPLICATION_SERVER_CONFIG is not updated with static membership Key: KAFKA-9568 URL: https://issues.apache.org/jira/browse/KAFKA-9568 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 2.4.0 Reporter: David J. Garcia A kstreams application with static membership, and StreamsConfg.APPLICATION_SERVER_CONFIG set, will NOT update old server config upon restart of application on new host. Steps to reproduce: # start two kstreams applications (with same consumer group) and enable static membership (and set application server config to :) # kill one of the applications and restart it on a new host(with new ip) before timeout ends (so that rebalancing doesn't occur). # the other kstreams application will now have an invalid application_server_config Possible fix: If an application restarts with a new host/identity..etc, it could trigger a "light-rebalance" where the other applications in the consumer group don't change partition assignments ,but instead just get their configuration updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9567) Docs and system tests for ZooKeeper 3.5.7 and KIP-515
[ https://issues.apache.org/jira/browse/KAFKA-9567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039197#comment-17039197 ] ASF GitHub Bot commented on KAFKA-9567: --- rondagostino commented on pull request #8132: KAFKA-9567: Docs, system tests for ZooKeeper 3.5.7 URL: https://github.com/apache/kafka/pull/8132 These changes depend on [KIP-515: Enable ZK client to use the new TLS supported authentication](https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication), which was only added to 2.5.0. The upgrade to ZooKeeper 3.5.7 was merged to both 2.5.0 and 2.4.1 via https://issues.apache.org/jira/browse/KAFKA-9515, but this change must only be merged to 2.5.0 (it will break the system tests if merged to 2.4.1). *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Docs and system tests for ZooKeeper 3.5.7 and KIP-515 > - > > Key: KAFKA-9567 > URL: https://issues.apache.org/jira/browse/KAFKA-9567 > Project: Kafka > Issue Type: Improvement >Affects Versions: 2.5.0 >Reporter: Ron Dagostino >Priority: Blocker > > These changes depend on [KIP-515: Enable ZK client to use the new TLS > supported > authentication|https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication], > which was only added to 2.5.0. The upgrade to ZooKeeper 3.5.7 was merged to > both 2.5.0 and 2.4.1 via https://issues.apache.org/jira/browse/KAFKA-9515, > but this change must only be merged to 2.5.0 (it will break the system tests > if merged to 2.4.1). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9567) Docs and system tests for ZooKeeper 3.5.7 and KIP-515
Ron Dagostino created KAFKA-9567: Summary: Docs and system tests for ZooKeeper 3.5.7 and KIP-515 Key: KAFKA-9567 URL: https://issues.apache.org/jira/browse/KAFKA-9567 Project: Kafka Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Ron Dagostino These changes depend on [KIP-515: Enable ZK client to use the new TLS supported authentication|https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication], which was only added to 2.5.0. The upgrade to ZooKeeper 3.5.7 was merged to both 2.5.0 and 2.4.1 via https://issues.apache.org/jira/browse/KAFKA-9515, but this change must only be merged to 2.5.0 (it will break the system tests if merged to 2.4.1). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9566) ProcessorContextImpl#forward throws NullPointerException if invoked from DeserializationExceptionHandler
Tomas Mi created KAFKA-9566:
---
Summary: ProcessorContextImpl#forward throws NullPointerException
if invoked from DeserializationExceptionHandler
Key: KAFKA-9566
URL: https://issues.apache.org/jira/browse/KAFKA-9566
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.2.0
Reporter: Tomas Mi
Hi, I am trying to implement custom DeserializationExceptionHandler which would
forward an exception to downstream processor(s), but
ProcessorContextImpl#forward throws a NullPointerException if invoked from this
custom handler.
Handler implementation:
{code:title=MyDeserializationExceptionHandler.java}
public class MyDeserializationExceptionHandler implements
DeserializationExceptionHandler {
@Override
public void configure(Map configs) {
}
@Override
public DeserializationHandlerResponse handle(ProcessorContext context,
ConsumerRecord record, Exception exception) {
context.forward(null, exception, To.child("error-processor"));
return DeserializationHandlerResponse.CONTINUE;
}
}
{code}
Handler is wired as default deserialization exception handler:
{code}
private TopologyTestDriver initializeTestDriver(StreamsBuilder
streamBuilder) {
Topology topology = streamBuilder.build();
Properties props = new Properties();
props.setProperty(StreamsConfig.APPLICATION_ID_CONFIG,
"my-test-application");
props.setProperty(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234");
props.setProperty(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
StreamsConfig.EXACTLY_ONCE);
props.setProperty(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
MyDeserializationExceptionHandler.class.getName());
return new TopologyTestDriver(topology, props);
}
{code}
Exception stacktrace:
{noformat}
org.apache.kafka.streams.errors.StreamsException: Fatal user code error in
deserialization error callback
at
org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:76)
at
org.apache.kafka.streams.processor.internals.RecordQueue.maybeUpdateTimestamp(RecordQueue.java:160)
at
org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:101)
at
org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:136)
at
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:742)
at
org.apache.kafka.streams.TopologyTestDriver.pipeInput(TopologyTestDriver.java:392)
...
Caused by: java.lang.NullPointerException
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:165)
at
MyDeserializationExceptionHandler.handle(NewExceptionHandlerTest.java:204)
at
org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:70)
... 33 more
{noformat}
Neither DeserializationExceptionHandler, nor ProcessorContext javadocs mention
that ProcessorContext#forward(...) must not be invoked from
DeserializationExceptionHandler, so I assume that this is a defect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-4090) JVM runs into OOM if (Java) client uses a SSL port without setting the security protocol
[
https://issues.apache.org/jira/browse/KAFKA-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039080#comment-17039080
]
Alexandre Dupriez edited comment on KAFKA-4090 at 2/18/20 1:25 PM:
---
Thanks [~belugabehr]. Shared on thread for[
KIP-498.|https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM?src=jira]
was (Author: adupriez):
Thanks [~belugabehr]. Shared on thread for[
KIP-498|https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM?src=jira]
> JVM runs into OOM if (Java) client uses a SSL port without setting the
> security protocol
>
>
> Key: KAFKA-4090
> URL: https://issues.apache.org/jira/browse/KAFKA-4090
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 0.9.0.1, 0.10.0.1, 2.1.0
>Reporter: Jaikiran Pai
>Assignee: Alexandre Dupriez
>Priority: Major
>
> Quoting from the mail thread that was sent to Kafka mailing list:
> {quote}
> We have been using Kafka 0.9.0.1 (server and Java client libraries). So far
> we had been using it with plaintext transport but recently have been
> considering upgrading to using SSL. It mostly works except that a
> mis-configured producer (and even consumer) causes a hard to relate
> OutOfMemory exception and thus causing the JVM in which the client is
> running, to go into a bad state. We can consistently reproduce that OOM very
> easily. We decided to check if this is something that is fixed in 0.10.0.1 so
> upgraded one of our test systems to that version (both server and client
> libraries) but still see the same issue. Here's how it can be easily
> reproduced
> 1. Enable SSL listener on the broker via server.properties, as per the Kafka
> documentation
> {code}
> listeners=PLAINTEXT://:9092,SSL://:9093
> ssl.keystore.location=
> ssl.keystore.password=pass
> ssl.key.password=pass
> ssl.truststore.location=
> ssl.truststore.password=pass
> {code}
> 2. Start zookeeper and kafka server
> 3. Create a "oom-test" topic (which will be used for these tests):
> {code}
> kafka-topics.sh --zookeeper localhost:2181 --create --topic oom-test
> --partitions 1 --replication-factor 1
> {code}
> 4. Create a simple producer which sends a single message to the topic via
> Java (new producer) APIs:
> {code}
> public class OOMTest {
> public static void main(final String[] args) throws Exception {
> final Properties kafkaProducerConfigs = new Properties();
> // NOTE: Intentionally use a SSL port without specifying
> security.protocol as SSL
>
> kafkaProducerConfigs.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
> "localhost:9093");
>
> kafkaProducerConfigs.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
>
> kafkaProducerConfigs.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
> try (KafkaProducer producer = new
> KafkaProducer<>(kafkaProducerConfigs)) {
> System.out.println("Created Kafka producer");
> final String topicName = "oom-test";
> final String message = "Hello OOM!";
> // send a message to the topic
> final Future recordMetadataFuture =
> producer.send(new ProducerRecord<>(topicName, message));
> final RecordMetadata sentRecordMetadata =
> recordMetadataFuture.get();
> System.out.println("Sent message '" + message + "' to topic '" +
> topicName + "'");
> }
> System.out.println("Tests complete");
> }
> }
> {code}
> Notice that the server URL is using a SSL endpoint localhost:9093 but isn't
> specifying any of the other necessary SSL configs like security.protocol.
> 5. For the sake of easily reproducing this issue run this class with a max
> heap size of 256MB (-Xmx256M). Running this code throws up the following
> OutOfMemoryError in one of the Sender threads:
> {code}
> 18:33:25,770 ERROR [KafkaThread] - Uncaught exception in
> kafka-producer-network-thread | producer-1:
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
> at
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
> at
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153)
> at
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134)
> at
[jira] [Comment Edited] (KAFKA-4090) JVM runs into OOM if (Java) client uses a SSL port without setting the security protocol
[
https://issues.apache.org/jira/browse/KAFKA-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039080#comment-17039080
]
Alexandre Dupriez edited comment on KAFKA-4090 at 2/18/20 1:25 PM:
---
Thanks [~belugabehr]. Shared on thread for[
KIP-498|https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM?src=jira]
was (Author: adupriez):
Thanks [~belugabehr]. Shared on thread for[
KIP-498|https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM?src=jira]]
> JVM runs into OOM if (Java) client uses a SSL port without setting the
> security protocol
>
>
> Key: KAFKA-4090
> URL: https://issues.apache.org/jira/browse/KAFKA-4090
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 0.9.0.1, 0.10.0.1, 2.1.0
>Reporter: Jaikiran Pai
>Assignee: Alexandre Dupriez
>Priority: Major
>
> Quoting from the mail thread that was sent to Kafka mailing list:
> {quote}
> We have been using Kafka 0.9.0.1 (server and Java client libraries). So far
> we had been using it with plaintext transport but recently have been
> considering upgrading to using SSL. It mostly works except that a
> mis-configured producer (and even consumer) causes a hard to relate
> OutOfMemory exception and thus causing the JVM in which the client is
> running, to go into a bad state. We can consistently reproduce that OOM very
> easily. We decided to check if this is something that is fixed in 0.10.0.1 so
> upgraded one of our test systems to that version (both server and client
> libraries) but still see the same issue. Here's how it can be easily
> reproduced
> 1. Enable SSL listener on the broker via server.properties, as per the Kafka
> documentation
> {code}
> listeners=PLAINTEXT://:9092,SSL://:9093
> ssl.keystore.location=
> ssl.keystore.password=pass
> ssl.key.password=pass
> ssl.truststore.location=
> ssl.truststore.password=pass
> {code}
> 2. Start zookeeper and kafka server
> 3. Create a "oom-test" topic (which will be used for these tests):
> {code}
> kafka-topics.sh --zookeeper localhost:2181 --create --topic oom-test
> --partitions 1 --replication-factor 1
> {code}
> 4. Create a simple producer which sends a single message to the topic via
> Java (new producer) APIs:
> {code}
> public class OOMTest {
> public static void main(final String[] args) throws Exception {
> final Properties kafkaProducerConfigs = new Properties();
> // NOTE: Intentionally use a SSL port without specifying
> security.protocol as SSL
>
> kafkaProducerConfigs.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
> "localhost:9093");
>
> kafkaProducerConfigs.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
>
> kafkaProducerConfigs.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
> try (KafkaProducer producer = new
> KafkaProducer<>(kafkaProducerConfigs)) {
> System.out.println("Created Kafka producer");
> final String topicName = "oom-test";
> final String message = "Hello OOM!";
> // send a message to the topic
> final Future recordMetadataFuture =
> producer.send(new ProducerRecord<>(topicName, message));
> final RecordMetadata sentRecordMetadata =
> recordMetadataFuture.get();
> System.out.println("Sent message '" + message + "' to topic '" +
> topicName + "'");
> }
> System.out.println("Tests complete");
> }
> }
> {code}
> Notice that the server URL is using a SSL endpoint localhost:9093 but isn't
> specifying any of the other necessary SSL configs like security.protocol.
> 5. For the sake of easily reproducing this issue run this class with a max
> heap size of 256MB (-Xmx256M). Running this code throws up the following
> OutOfMemoryError in one of the Sender threads:
> {code}
> 18:33:25,770 ERROR [KafkaThread] - Uncaught exception in
> kafka-producer-network-thread | producer-1:
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
> at
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
> at
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153)
> at
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134)
> at
[jira] [Commented] (KAFKA-4090) JVM runs into OOM if (Java) client uses a SSL port without setting the security protocol
[
https://issues.apache.org/jira/browse/KAFKA-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039080#comment-17039080
]
Alexandre Dupriez commented on KAFKA-4090:
--
Thanks [~belugabehr]. Shared on thread for
[[KIP-498|https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM?src=jira]|https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM?src=jira]
> JVM runs into OOM if (Java) client uses a SSL port without setting the
> security protocol
>
>
> Key: KAFKA-4090
> URL: https://issues.apache.org/jira/browse/KAFKA-4090
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 0.9.0.1, 0.10.0.1, 2.1.0
>Reporter: Jaikiran Pai
>Assignee: Alexandre Dupriez
>Priority: Major
>
> Quoting from the mail thread that was sent to Kafka mailing list:
> {quote}
> We have been using Kafka 0.9.0.1 (server and Java client libraries). So far
> we had been using it with plaintext transport but recently have been
> considering upgrading to using SSL. It mostly works except that a
> mis-configured producer (and even consumer) causes a hard to relate
> OutOfMemory exception and thus causing the JVM in which the client is
> running, to go into a bad state. We can consistently reproduce that OOM very
> easily. We decided to check if this is something that is fixed in 0.10.0.1 so
> upgraded one of our test systems to that version (both server and client
> libraries) but still see the same issue. Here's how it can be easily
> reproduced
> 1. Enable SSL listener on the broker via server.properties, as per the Kafka
> documentation
> {code}
> listeners=PLAINTEXT://:9092,SSL://:9093
> ssl.keystore.location=
> ssl.keystore.password=pass
> ssl.key.password=pass
> ssl.truststore.location=
> ssl.truststore.password=pass
> {code}
> 2. Start zookeeper and kafka server
> 3. Create a "oom-test" topic (which will be used for these tests):
> {code}
> kafka-topics.sh --zookeeper localhost:2181 --create --topic oom-test
> --partitions 1 --replication-factor 1
> {code}
> 4. Create a simple producer which sends a single message to the topic via
> Java (new producer) APIs:
> {code}
> public class OOMTest {
> public static void main(final String[] args) throws Exception {
> final Properties kafkaProducerConfigs = new Properties();
> // NOTE: Intentionally use a SSL port without specifying
> security.protocol as SSL
>
> kafkaProducerConfigs.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
> "localhost:9093");
>
> kafkaProducerConfigs.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
>
> kafkaProducerConfigs.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
> try (KafkaProducer producer = new
> KafkaProducer<>(kafkaProducerConfigs)) {
> System.out.println("Created Kafka producer");
> final String topicName = "oom-test";
> final String message = "Hello OOM!";
> // send a message to the topic
> final Future recordMetadataFuture =
> producer.send(new ProducerRecord<>(topicName, message));
> final RecordMetadata sentRecordMetadata =
> recordMetadataFuture.get();
> System.out.println("Sent message '" + message + "' to topic '" +
> topicName + "'");
> }
> System.out.println("Tests complete");
> }
> }
> {code}
> Notice that the server URL is using a SSL endpoint localhost:9093 but isn't
> specifying any of the other necessary SSL configs like security.protocol.
> 5. For the sake of easily reproducing this issue run this class with a max
> heap size of 256MB (-Xmx256M). Running this code throws up the following
> OutOfMemoryError in one of the Sender threads:
> {code}
> 18:33:25,770 ERROR [KafkaThread] - Uncaught exception in
> kafka-producer-network-thread | producer-1:
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
> at
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
> at
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153)
> at
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
> at
[jira] [Comment Edited] (KAFKA-4090) JVM runs into OOM if (Java) client uses a SSL port without setting the security protocol
[
https://issues.apache.org/jira/browse/KAFKA-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039080#comment-17039080
]
Alexandre Dupriez edited comment on KAFKA-4090 at 2/18/20 1:25 PM:
---
Thanks [~belugabehr]. Shared on thread for[
KIP-498|https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM?src=jira]]
was (Author: adupriez):
Thanks [~belugabehr]. Shared on thread for
[[KIP-498|https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM?src=jira]|https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM?src=jira]
> JVM runs into OOM if (Java) client uses a SSL port without setting the
> security protocol
>
>
> Key: KAFKA-4090
> URL: https://issues.apache.org/jira/browse/KAFKA-4090
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 0.9.0.1, 0.10.0.1, 2.1.0
>Reporter: Jaikiran Pai
>Assignee: Alexandre Dupriez
>Priority: Major
>
> Quoting from the mail thread that was sent to Kafka mailing list:
> {quote}
> We have been using Kafka 0.9.0.1 (server and Java client libraries). So far
> we had been using it with plaintext transport but recently have been
> considering upgrading to using SSL. It mostly works except that a
> mis-configured producer (and even consumer) causes a hard to relate
> OutOfMemory exception and thus causing the JVM in which the client is
> running, to go into a bad state. We can consistently reproduce that OOM very
> easily. We decided to check if this is something that is fixed in 0.10.0.1 so
> upgraded one of our test systems to that version (both server and client
> libraries) but still see the same issue. Here's how it can be easily
> reproduced
> 1. Enable SSL listener on the broker via server.properties, as per the Kafka
> documentation
> {code}
> listeners=PLAINTEXT://:9092,SSL://:9093
> ssl.keystore.location=
> ssl.keystore.password=pass
> ssl.key.password=pass
> ssl.truststore.location=
> ssl.truststore.password=pass
> {code}
> 2. Start zookeeper and kafka server
> 3. Create a "oom-test" topic (which will be used for these tests):
> {code}
> kafka-topics.sh --zookeeper localhost:2181 --create --topic oom-test
> --partitions 1 --replication-factor 1
> {code}
> 4. Create a simple producer which sends a single message to the topic via
> Java (new producer) APIs:
> {code}
> public class OOMTest {
> public static void main(final String[] args) throws Exception {
> final Properties kafkaProducerConfigs = new Properties();
> // NOTE: Intentionally use a SSL port without specifying
> security.protocol as SSL
>
> kafkaProducerConfigs.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
> "localhost:9093");
>
> kafkaProducerConfigs.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
>
> kafkaProducerConfigs.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
> try (KafkaProducer producer = new
> KafkaProducer<>(kafkaProducerConfigs)) {
> System.out.println("Created Kafka producer");
> final String topicName = "oom-test";
> final String message = "Hello OOM!";
> // send a message to the topic
> final Future recordMetadataFuture =
> producer.send(new ProducerRecord<>(topicName, message));
> final RecordMetadata sentRecordMetadata =
> recordMetadataFuture.get();
> System.out.println("Sent message '" + message + "' to topic '" +
> topicName + "'");
> }
> System.out.println("Tests complete");
> }
> }
> {code}
> Notice that the server URL is using a SSL endpoint localhost:9093 but isn't
> specifying any of the other necessary SSL configs like security.protocol.
> 5. For the sake of easily reproducing this issue run this class with a max
> heap size of 256MB (-Xmx256M). Running this code throws up the following
> OutOfMemoryError in one of the Sender threads:
> {code}
> 18:33:25,770 ERROR [KafkaThread] - Uncaught exception in
> kafka-producer-network-thread | producer-1:
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
> at
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
> at
>
[jira] [Commented] (KAFKA-9543) Consumer offset reset after new segment rolling
[
https://issues.apache.org/jira/browse/KAFKA-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039062#comment-17039062
]
Alexandre Dupriez commented on KAFKA-9543:
--
Interesting. Does the screenshot attached represent consumer lag? Do you have a
self-contained reproduction use-case?
> Consumer offset reset after new segment rolling
> ---
>
> Key: KAFKA-9543
> URL: https://issues.apache.org/jira/browse/KAFKA-9543
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Rafał Boniecki
>Priority: Major
> Attachments: Untitled.png
>
>
> After upgrade from kafka 2.1.1 to 2.4.0, I'm experiencing unexpected consumer
> offset resets.
> Consumer:
> {code:java}
> 2020-02-12T11:12:58.402+01:00 hostname 4a2a39a35a02
> [2020-02-12T11:12:58,402][INFO
> ][org.apache.kafka.clients.consumer.internals.Fetcher] [Consumer
> clientId=logstash-1, groupId=logstash] Fetch offset 1632750575 is out of
> range for partition stats-5, resetting offset
> {code}
> Broker:
> {code:java}
> 2020-02-12 11:12:58:400 CET INFO
> [data-plane-kafka-request-handler-1][kafka.log.Log] [Log partition=stats-5,
> dir=/kafka4/data] Rolled new log segment at offset 1632750565 in 2 ms.{code}
> All resets are perfectly correlated to rolling new segments at the broker -
> segment is rolled first, then, couple of ms later, reset on the consumer
> occurs. Attached is grafana graph with consumer lag per partition. All sudden
> spikes in lag are offset resets due to this bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9335) java.lang.IllegalArgumentException: Number of partitions must be at least 1.
[
https://issues.apache.org/jira/browse/KAFKA-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039049#comment-17039049
]
Vijay Veeramani commented on KAFKA-9335:
[~bchen225242] thanks for looking into this... Is there any work around for
this defect, till 2.4.1 release becomes available?
> java.lang.IllegalArgumentException: Number of partitions must be at least 1.
>
>
> Key: KAFKA-9335
> URL: https://issues.apache.org/jira/browse/KAFKA-9335
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0
>Reporter: Nitay Kufert
>Assignee: Boyang Chen
>Priority: Blocker
> Labels: bug
> Fix For: 2.4.1
>
>
> Hey,
> When trying to upgrade our Kafka streams client to 2.4.0 (from 2.3.1) we
> encountered the following exception:
> {code:java}
> java.lang.IllegalArgumentException: Number of partitions must be at least 1.
> {code}
> It's important to notice that the exact same code works just fine at 2.3.1.
>
> I have created a "toy" example which reproduces this exception:
> [https://gist.github.com/nitayk/50da33b7bcce19ad0a7f8244d309cb8f]
> and I would love to get some insight regarding why its happening / ways to
> get around it
>
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9319) Run some system tests using TLSv1.3
[ https://issues.apache.org/jira/browse/KAFKA-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039048#comment-17039048 ] Rajini Sivaram commented on KAFKA-9319: --- [~nizhikov] We want to make it possible to optionally run system tests with TLSv1.3 when the tests are run with Java version 11 or above. This is required to make sure there are no regressions in future, without having to change the test code as you did this time. But that change can be done under the KIP for KAFKA-9320. Just to confirm - have all the system tests using SSL passed with TLSv1.3 in your local run? There were some failures in the output you pasted earlier. Have you rerun those? If all the tests using SSL have passed with TLSv1.3, feel free to close this JIRA. > Run some system tests using TLSv1.3 > --- > > Key: KAFKA-9319 > URL: https://issues.apache.org/jira/browse/KAFKA-9319 > Project: Kafka > Issue Type: Test >Reporter: Rajini Sivaram >Assignee: Nikolay Izhikov >Priority: Major > Fix For: 2.5.0 > > > KAFKA-7251 enables TLSv1.3 for Kafka. We should get some system tests to run > using TLSv1.3. Since TLSv1.3 is only supported from Java 11 onwards, we need > a system test build that runs with JDK 11 to enable these tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9319) Run some system tests using TLSv1.3
[ https://issues.apache.org/jira/browse/KAFKA-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039022#comment-17039022 ] Nikolay Izhikov commented on KAFKA-9319: [~rsivaram] Thanks for the answers. It seems the results of my tests show that if TLSv1.3 enabled then system tests are still passed. So I think we can resolve this issue and process with the KAFKA-9320, isn't it? > Run some system tests using TLSv1.3 > --- > > Key: KAFKA-9319 > URL: https://issues.apache.org/jira/browse/KAFKA-9319 > Project: Kafka > Issue Type: Test >Reporter: Rajini Sivaram >Assignee: Nikolay Izhikov >Priority: Major > Fix For: 2.5.0 > > > KAFKA-7251 enables TLSv1.3 for Kafka. We should get some system tests to run > using TLSv1.3. Since TLSv1.3 is only supported from Java 11 onwards, we need > a system test build that runs with JDK 11 to enable these tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9530) Flaky Test kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout
[
https://issues.apache.org/jira/browse/KAFKA-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039015#comment-17039015
]
Stanislav Kozlovski commented on KAFKA-9530:
I managed to reproduce the failure with 1 run (out of 96) locally - the issue
is that DisconnectException gets raised:
{code:java}
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.DisconnectException: Cancelled findCoordinator
request with correlation id 3 due to node 0 being disconnected
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$describeConsumerGroups$1(ConsumerGroupCommand.scala:497)
...
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeConsumerGroups(ConsumerGroupCommand.scala:496)
...
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.collectGroupsMembers(ConsumerGroupCommand.scala:552)
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeGroups(ConsumerGroupCommand.scala:318)
kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout(DescribeConsumerGroupTest.scala:582){code}
---
It seems like the AdminClient handles timed out calls by disconnecting
([https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L1075])
- this adds a disconnected flag to the response which we then handle via a
DisconnectedException
([https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L1126])
> Flaky Test
> kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout
>
>
> Key: KAFKA-9530
> URL: https://issues.apache.org/jira/browse/KAFKA-9530
> Project: Kafka
> Issue Type: Test
> Components: core
>Reporter: Bill Bejeck
>Priority: Major
> Labels: flaky-test, test
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/4570/testReport/junit/kafka.admin/DescribeConsumerGroupTest/testDescribeGroupWithShortInitializationTimeout/]
>
> {noformat}
> Error Messagejava.lang.AssertionError: assertion
> failedStacktracejava.lang.AssertionError: assertion failed
> at scala.Predef$.assert(Predef.scala:267)
> at
> kafka.admin.DescribeConsumerGroupTest.testDescribeGroupWithShortInitializationTimeout(DescribeConsumerGroupTest.scala:585)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
>
[jira] [Commented] (KAFKA-9319) Run some system tests using TLSv1.3
[ https://issues.apache.org/jira/browse/KAFKA-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039001#comment-17039001 ] Rajini Sivaram commented on KAFKA-9319: --- [~nizhikov] Yes, that is correct. We can enable TLSv1.3 in `ssl.enabled.protocols`, but we cannot disable TLSv1.2 or make TLSv1.3 the default protocol in `ssl.protocol`. > Run some system tests using TLSv1.3 > --- > > Key: KAFKA-9319 > URL: https://issues.apache.org/jira/browse/KAFKA-9319 > Project: Kafka > Issue Type: Test >Reporter: Rajini Sivaram >Assignee: Nikolay Izhikov >Priority: Major > Fix For: 2.5.0 > > > KAFKA-7251 enables TLSv1.3 for Kafka. We should get some system tests to run > using TLSv1.3. Since TLSv1.3 is only supported from Java 11 onwards, we need > a system test build that runs with JDK 11 to enable these tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9319) Run some system tests using TLSv1.3
[ https://issues.apache.org/jira/browse/KAFKA-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038974#comment-17038974 ] Nikolay Izhikov commented on KAFKA-9319: [~rsivaram] > No, we cannot require users to run on Java 11 since we need to continue > support for Java 8 Does this mean that we can't complete KAFKA-9320 for now and enable TLSv1.3 by default and disable TLSv1.2? > Run some system tests using TLSv1.3 > --- > > Key: KAFKA-9319 > URL: https://issues.apache.org/jira/browse/KAFKA-9319 > Project: Kafka > Issue Type: Test >Reporter: Rajini Sivaram >Assignee: Nikolay Izhikov >Priority: Major > Fix For: 2.5.0 > > > KAFKA-7251 enables TLSv1.3 for Kafka. We should get some system tests to run > using TLSv1.3. Since TLSv1.3 is only supported from Java 11 onwards, we need > a system test build that runs with JDK 11 to enable these tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9565) Implementation of Tiered Storage SPI to integrate with S3
[ https://issues.apache.org/jira/browse/KAFKA-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Yurchenko reassigned KAFKA-9565: - Assignee: Ivan Yurchenko > Implementation of Tiered Storage SPI to integrate with S3 > - > > Key: KAFKA-9565 > URL: https://issues.apache.org/jira/browse/KAFKA-9565 > Project: Kafka > Issue Type: Sub-task >Reporter: Alexandre Dupriez >Assignee: Ivan Yurchenko >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9319) Run some system tests using TLSv1.3
[ https://issues.apache.org/jira/browse/KAFKA-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038970#comment-17038970 ] Rajini Sivaram commented on KAFKA-9319: --- [~nizhikov] No, we cannot require users to run on Java 11 since we need to continue support for Java 8. For integration tests, we run some tests with TLSv1.3 when running with Java 11. Those tests run with TLSv1.2 on Java 8. We should do something similar for system tests as well. We could convert a subset of the tests to choose TLSv1.3 at runtime if the Java version is 11 or above. > Run some system tests using TLSv1.3 > --- > > Key: KAFKA-9319 > URL: https://issues.apache.org/jira/browse/KAFKA-9319 > Project: Kafka > Issue Type: Test >Reporter: Rajini Sivaram >Assignee: Nikolay Izhikov >Priority: Major > Fix For: 2.5.0 > > > KAFKA-7251 enables TLSv1.3 for Kafka. We should get some system tests to run > using TLSv1.3. Since TLSv1.3 is only supported from Java 11 onwards, we need > a system test build that runs with JDK 11 to enable these tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9565) Implementation of Tiered Storage SPI to integrate with S3
Alexandre Dupriez created KAFKA-9565: Summary: Implementation of Tiered Storage SPI to integrate with S3 Key: KAFKA-9565 URL: https://issues.apache.org/jira/browse/KAFKA-9565 Project: Kafka Issue Type: Sub-task Reporter: Alexandre Dupriez -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9376) Plugin class loader not found using MM2
[
https://issues.apache.org/jira/browse/KAFKA-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038934#comment-17038934
]
candan commented on KAFKA-9376:
---
Hello;
We also encountred the same problem? Did you find any clue.
> Plugin class loader not found using MM2
> ---
>
> Key: KAFKA-9376
> URL: https://issues.apache.org/jira/browse/KAFKA-9376
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 2.4.0
>Reporter: Sinóros-Szabó Péter
>Priority: Minor
>
> I am using MM2 (release 2.4.0 with scala 2.12) I geta bunch of classloader
> errors. MM2 seems to be working, but I do not know if all of it components
> are working as expected as this is the first time I use MM2.
> I run MM2 with the following command:
> {code:java}
> ./bin/connect-mirror-maker.sh config/connect-mirror-maker.properties
> {code}
> Errors are:
> {code:java}
> [2020-01-07 15:06:17,892] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,889] ERROR Plugin class loader for connector:
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@6ebf0f36
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165)
> [2020-01-07 15:06:17,904] INFO ConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
> (org.apache.kafka.connect.runtime.ConnectorConfig:347)
> [2020-01-07 15:06:17,904] INFO EnrichedConnectorConfig values:
> config.action.reload = restart
> connector.class = org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
> errors.log.enable = false
> errors.log.include.messages = false
> errors.retry.delay.max.ms = 6
> errors.retry.timeout = 0
> errors.tolerance = none
> header.converter = null
> key.converter = null
> name = MirrorHeartbeatConnector
> tasks.max = 1
> transforms = []
> value.converter = null
>
> (org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
> [2020-01-07 15:06:17,905] INFO TaskConfig values:
> task.class = class org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.TaskConfig:347)
> [2020-01-07 15:06:17,905] INFO Instantiated task MirrorHeartbeatConnector-0
> with version 1 of type org.apache.kafka.connect.mirror.MirrorHeartbeatTask
> (org.apache.kafka.connect.runtime.Worker:434){code}
> After a while, these errors are not logged any more.
> Config is:
> {code:java}
> clusters = eucmain, euwbackup
> eucmain.bootstrap.servers = kafka1:9092,kafka2:9092
> euwbackup.bootstrap.servers = 172.30.197.203:9092,172.30.213.104:9092
> eucmain->euwbackup.enabled = true
> eucmain->euwbackup.topics = .*
> eucmain->euwbackup.topics.blacklist = ^(kafka|kmf|__|pricing).*
> eucmain->euwbackup.rename.topics = false
> rename.topics = false
> eucmain->euwbackup.sync.topic.acls.enabled = false
> sync.topic.acls.enabled = false{code}
> Using OpenJDK 8 or 11, I get the same error.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9512) Flaky Test LagFetchIntegrationTest.shouldFetchLagsDuringRestoration
[
https://issues.apache.org/jira/browse/KAFKA-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038870#comment-17038870
]
Bruno Cadonna commented on KAFKA-9512:
--
[~vinoth][~guozhang] Can we close this ticket?
> Flaky Test LagFetchIntegrationTest.shouldFetchLagsDuringRestoration
> ---
>
> Key: KAFKA-9512
> URL: https://issues.apache.org/jira/browse/KAFKA-9512
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.5.0
>Reporter: Matthias J. Sax
>Assignee: Vinoth Chandar
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.5.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/497/testReport/junit/org.apache.kafka.streams.integration/LagFetchIntegrationTest/shouldFetchLagsDuringRestoration/]
> {quote}java.lang.NullPointerException at
> org.apache.kafka.streams.integration.LagFetchIntegrationTest.shouldFetchLagsDuringRestoration(LagFetchIntegrationTest.java:306){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9540) Application getting "Could not find the standby task 0_4 while closing it" error
[
https://issues.apache.org/jira/browse/KAFKA-9540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038868#comment-17038868
]
Bruno Cadonna commented on KAFKA-9540:
--
[~ableegoldman] [~guozhang] Can we close this ticket?
> Application getting "Could not find the standby task 0_4 while closing it"
> error
>
>
> Key: KAFKA-9540
> URL: https://issues.apache.org/jira/browse/KAFKA-9540
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Badai Aqrandista
>Priority: Minor
> Fix For: 2.4.1
>
>
> Because of this the following line, there is a possibility that some standby
> tasks might not be created:
> https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L436
> Then causing this line to not adding the task to standby task list:
> https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L299
> But this line assumes that all standby tasks are to be created and add it to
> the standby list:
> https://github.com/apache/kafka/blob/2.4.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java#L168
> This results in user getting this error message on the next
> PARTITION_ASSIGNMENT state:
> {noformat}
> Could not find the standby task 0_4 while closing it
> (org.apache.kafka.streams.processor.internals.AssignedStandbyTasks:74)
> {noformat}
> But the harm caused by this issue is minimal: No standby task for some
> partitions. And it is recreated on the next rebalance anyway. So, I suggest
> lowering this message to WARN. Or probably check to WARN when standby task
> could not be created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
