[jira] [Created] (KAFKA-8820) Use Admin API of Replica Reassignment in CLI tools
Gwen Shapira created KAFKA-8820: --- Summary: Use Admin API of Replica Reassignment in CLI tools Key: KAFKA-8820 URL: https://issues.apache.org/jira/browse/KAFKA-8820 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Steve Rodrigues KIP-455 and KAFKA-8345 add a protocol and AdminAPI that will be used for replica reassignments. We need to update the reassignment tool to use this new API rather than work with ZK directly. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-8737) TaskMigrated Exception while rebalancing kafka streams
[ https://issues.apache.org/jira/browse/KAFKA-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910840#comment-16910840 ] Guozhang Wang commented on KAFKA-8737: -- Hi [~bbejeck] have you looked at this issue? How should we fix it? > TaskMigrated Exception while rebalancing kafka streams > -- > > Key: KAFKA-8737 > URL: https://issues.apache.org/jira/browse/KAFKA-8737 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 1.0.0, 1.0.1 > Environment: 20 partitions > 1 topic > 8 Streamer service > topic-region-1 9 7841726 8236017 > 394291 > streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/ > > streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer > topic-region-1 15 7421710 7467666 > 45956 > streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/ > > streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer > topic-region-1 19 7737360 8120611 > 383251 > streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/ > > streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer > topic-region-1 >Reporter: KUMAR >Assignee: Bill Bejeck >Priority: Major > > Kafka streams throws following exception while restart of a stream client > service - > o.a.k.s.p.internals.StreamThread.? - stream-thread > [streams-subscriberstopic-region-1-32d968e3-f892-4772-a7a4-6f684d7e43c9-StreamThread-1] > Detected a task that got migrated to another thread. This implies that this > thread missed a rebalance and dropped out of the consumer group. Trying to > rejoin the consumer group now. > org.apache.kafka.streams.errors.TaskMigratedException: Log end offset of > topic-region-1-12 should not change while restoring: old end offset 6286727, > current offset 6380997 > > Kafka version is 1.0.0 and we have back merged the fix for KIP-6269- > [https://github.com/apache/kafka/pull/4300/files#|https://github.com/apache/kafka/pull/4300/files] > However we observe that there seems to be an issue in rebalance when > "auto.offset.reset" is configured as "latest". Based on log analysis we see > following behavior - > # StreamThread starts a restore consumer > # While Fetching it gets offset out of range > o.a.k.c.consumer.internals.Fetcher.? - [Consumer > clientId=streams-subscriberstopic-region-1-11b2d7fb-11ce-4b0b-a40a-388d3c7b6bc9-StreamThread-1-restore- > consumer, groupId=] Fetch READ_UNCOMMITTED at offset 246431 for partition > topic-region-1-12 returned fetch data (error=OFFSET_OUT_OF_RANGE, > highWaterMark=-1, lastStableOffset = -1, logStartOffset = -1, > abortedTransactions = null, recordsSizeInBytes=0) > # Fetcher tries to reset the offset > # While reset the offset it appears it is changing the offset position and > causing TaskMigrated exception > Above test repeated with "auto.offset.reset" is configured as "earliest" does > not throw any TaskMigrated exception as in earliest case we are not reseting > the restore consumer position. > > Please let us know if this is possible and if a fix would be needed for the > offset reset piece when set to latest. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-4212) Add a key-value store that is a TTL persistent cache
[
https://issues.apache.org/jira/browse/KAFKA-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910835#comment-16910835
]
Guozhang Wang commented on KAFKA-4212:
--
Hello James, I'd also recommend you checking if the FIFO compaction would work
for you; if it does not and you still want to add the TTL for rocksDB, I think
your current proposal in the PR:
{code}
persistentKeyValueTtlStore(final String name, final int ttlInSeconds)
{code}
looks good to me, we can go through a KIP with this proposal.
> Add a key-value store that is a TTL persistent cache
>
>
> Key: KAFKA-4212
> URL: https://issues.apache.org/jira/browse/KAFKA-4212
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Priority: Major
> Labels: api
>
> Some jobs needs to maintain as state a large set of key-values for some
> period of time. I.e. they need to maintain a TTL cache of values potentially
> larger than memory.
> Currently Kafka Streams provides non-windowed and windowed key-value stores.
> Neither is an exact fit to this use case.
> The {{RocksDBStore}}, a {{KeyValueStore}}, stores one value per key as
> required, but does not support expiration. The TTL option of RocksDB is
> explicitly not used.
> The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment
> dropping, but it stores multiple items per key, based on their timestamp.
> But this store can be repurposed as a cache by fetching the items in reverse
> chronological order and returning the first item found.
> KAFKA-2594 introduced a fixed-capacity in-memory LRU caching store, but here
> we desire a variable-capacity memory-overflowing TTL caching store.
> Although {{RocksDBWindowsStore}} can be repurposed as a cache, it would be
> useful to have an official and proper TTL cache API and implementation.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-4650) Improve test coverage org.apache.kafka.streams.kstream.internals
[
https://issues.apache.org/jira/browse/KAFKA-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910833#comment-16910833
]
Matthias J. Sax commented on KAFKA-4650:
Thanks for you interest and welcome to the Kafka community!
I assigned the ticket to you and also added you to the list of contributors.
You can now self-assign tickets, too.
This ticket is somewhat old, hence the list of classes might be outdate, but I
am sure that we have many test gaps in package `internals`. I would recommend
to start a single PR for each class you want to add tests for. It makes it
easier to review and we can merge PRs faster. Let us know if you have any
questions.
> Improve test coverage org.apache.kafka.streams.kstream.internals
>
>
> Key: KAFKA-4650
> URL: https://issues.apache.org/jira/browse/KAFKA-4650
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Damian Guy
>Assignee: Hanumanth Bandi
>Priority: Minor
> Labels: newbie
>
> Lots of classes have little or no coverage at all, i.e.,
> {{KTableAggregate.KTableAggregateValueGetter}}
> {{KTableKTableRightJoin.KTableKTableRightJoinValueGetterSupplier}}
> {{KStreamAggregate.KStreamAggregateValueGetter}}
> {{KStreamReduce.KStreamReduceValueGetter}}
> {{KStreamWindowReduce.new KTableValueGetterSupplier}}
> {{KTableAggregate.new KTableValueGetterSupplier}}
> {{KTableRepartitionMap.new KTableValueGetterSupplier}}
> {{KTableKTableRightJoin.KTableKTableRightJoinValueGetter}}
> {{KTableKTableLeftJoinValueGetter}}
> {{KStreamWindowReduce.KStreamWindowReduceValueGetter}}
> {{TimeWindow}}
> {{ChangedSerializer}}
> {{UnlimitedWindow}}
> {{WindowedDeserializer}}
> {{KStreamSessionWindowAggregate.KTableSessionWindowValueGetter}}
> {{KTableRepartitionMap}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (KAFKA-4650) Improve test coverage org.apache.kafka.streams.kstream.internals
[
https://issues.apache.org/jira/browse/KAFKA-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reassigned KAFKA-4650:
--
Assignee: Hanumanth Bandi
> Improve test coverage org.apache.kafka.streams.kstream.internals
>
>
> Key: KAFKA-4650
> URL: https://issues.apache.org/jira/browse/KAFKA-4650
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Damian Guy
>Assignee: Hanumanth Bandi
>Priority: Minor
> Labels: newbie
>
> Lots of classes have little or no coverage at all, i.e.,
> {{KTableAggregate.KTableAggregateValueGetter}}
> {{KTableKTableRightJoin.KTableKTableRightJoinValueGetterSupplier}}
> {{KStreamAggregate.KStreamAggregateValueGetter}}
> {{KStreamReduce.KStreamReduceValueGetter}}
> {{KStreamWindowReduce.new KTableValueGetterSupplier}}
> {{KTableAggregate.new KTableValueGetterSupplier}}
> {{KTableRepartitionMap.new KTableValueGetterSupplier}}
> {{KTableKTableRightJoin.KTableKTableRightJoinValueGetter}}
> {{KTableKTableLeftJoinValueGetter}}
> {{KStreamWindowReduce.KStreamWindowReduceValueGetter}}
> {{TimeWindow}}
> {{ChangedSerializer}}
> {{UnlimitedWindow}}
> {{WindowedDeserializer}}
> {{KStreamSessionWindowAggregate.KTableSessionWindowValueGetter}}
> {{KTableRepartitionMap}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8818) CreatePartitions Request protocol documentation
[
https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fábio Silva updated KAFKA-8818:
---
Priority: Major (was: Minor)
> CreatePartitions Request protocol documentation
> ---
>
> Key: KAFKA-8818
> URL: https://issues.apache.org/jira/browse/KAFKA-8818
> Project: Kafka
> Issue Type: Bug
> Components: documentation
>Reporter: Fábio Silva
>Priority: Major
> Labels: documentation, protocol-documentation
>
> CreatePartitions Request protocol documentation contains a invalid type
> ARRAY(INT32) (assignment field), it must be INT32.
> Wrong:
> {code:java}
> assignment => ARRAY(INT32){code}
> Correct:
> {code:java}
> assignment => INT32
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-8522) Tombstones can survive forever
[ https://issues.apache.org/jira/browse/KAFKA-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910793#comment-16910793 ] Jun Rao commented on KAFKA-8522: [~Yohan123], I was thinking the new checkpoint file will be under disk_x/topic_y/z for each partition. We will deprecate the old checkpoint file under disk_x/. > Tombstones can survive forever > -- > > Key: KAFKA-8522 > URL: https://issues.apache.org/jira/browse/KAFKA-8522 > Project: Kafka > Issue Type: Improvement > Components: log cleaner >Reporter: Evelyn Bayes >Priority: Minor > > This is a bit grey zone as to whether it's a "bug" but it is certainly > unintended behaviour. > > Under specific conditions tombstones effectively survive forever: > * Small amount of throughput; > * min.cleanable.dirty.ratio near or at 0; and > * Other parameters at default. > What happens is all the data continuously gets cycled into the oldest > segment. Old records get compacted away, but the new records continuously > update the timestamp of the oldest segment reseting the countdown for > deleting tombstones. > So tombstones build up in the oldest segment forever. > > While you could "fix" this by reducing the segment size, this can be > undesirable as a sudden change in throughput could cause a dangerous number > of segments to be created. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-8522) Tombstones can survive forever
[ https://issues.apache.org/jira/browse/KAFKA-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910791#comment-16910791 ] Dhruvil Shah commented on KAFKA-8522: - [~Yohan123] yes, the idea would be to have one checkpoint file per partition stored alongside the log data. Each log directory can contain multiple partitions like you said. > Tombstones can survive forever > -- > > Key: KAFKA-8522 > URL: https://issues.apache.org/jira/browse/KAFKA-8522 > Project: Kafka > Issue Type: Improvement > Components: log cleaner >Reporter: Evelyn Bayes >Priority: Minor > > This is a bit grey zone as to whether it's a "bug" but it is certainly > unintended behaviour. > > Under specific conditions tombstones effectively survive forever: > * Small amount of throughput; > * min.cleanable.dirty.ratio near or at 0; and > * Other parameters at default. > What happens is all the data continuously gets cycled into the oldest > segment. Old records get compacted away, but the new records continuously > update the timestamp of the oldest segment reseting the countdown for > deleting tombstones. > So tombstones build up in the oldest segment forever. > > While you could "fix" this by reducing the segment size, this can be > undesirable as a sudden change in throughput could cause a dangerous number > of segments to be created. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8819) Plugin path for converters not working as intended
[ https://issues.apache.org/jira/browse/KAFKA-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magesh kumar Nandakumar updated KAFKA-8819: --- Description: KafakConnect allows all plugins to be available via a plugin path mechanism. This allows for classpath isolation. This is not working as designed under the following circumstances for Converters I have 2 directories under plugin path `connector1` and `connector2`. I intend to use AvroConverter and its available in both the plugin directories. Under these circumstances, the Worker attempts to create the Converter available in the plugin director first which should ideally be deterministic but it's not because of the following reasons:- [https://github.com/apache/kafka/blob/aa4ba8eee8e6f52a9d80a98fb2530b5bcc1b9a11/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L421] would lead to all configs of type Class to be loaded and this would mean that they are not loaded in the context of the connectors plugin loader. IIUC, the current loaded would be the DelegatingClassLoader. This would mean that the AvroConverter could potentially be loaded from connector2 plugin path while loading the class. This should be made deterministic as intended. Also, there could be instances where the Converter itself could potentially be using ServiceLoader and for that to work when the Converter is created & configured the current class loader should be set to the one corresponding to the converter. It might also be better to load all the config in the connector config in the context of connector's plugin loader to avoid issues with classes referenced across class loader. was: KafakConnect allows all plugins to be available via a plugin path mechanism. This allows for classpath isolation. This is not working as designed under the following circumstances for Converters I have 2 directories under plugin path `connector1` and `connector2`. I intend to use AvroConverter and its available in both the plugin directories. Under these circumstances, the Worker attempts to create the Converter available in the plugin director first which should ideally be deterministic but it's not because of the following reasons:- [https://github.com/apache/kafka/blob/aa4ba8eee8e6f52a9d80a98fb2530b5bcc1b9a11/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L421] would lead to all configs of type Class to be loaded and this would mean that they are not loaded in the context of the connectors plugin loader. IIUC, the current loaded would be the DelegatingClassLoader . This would mean that the AvroConverter could potentially be loaded from connector2 plugin path while loading the class. This should be made deterministic as intended. Also, there could be instances where the Converter itself could potentially be using ServiceLoader and for that to work when the Converter is created & configured the current class loader should be set to the one corresponding to the converter. > Plugin path for converters not working as intended > -- > > Key: KAFKA-8819 > URL: https://issues.apache.org/jira/browse/KAFKA-8819 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Magesh kumar Nandakumar >Assignee: Magesh kumar Nandakumar >Priority: Major > > KafakConnect allows all plugins to be available via a plugin path mechanism. > This allows for classpath isolation. This is not working as designed under > the following circumstances for Converters > > I have 2 directories under plugin path `connector1` and `connector2`. I > intend to use AvroConverter and its available in both the plugin directories. > Under these circumstances, the Worker attempts to create the Converter > available in the plugin director first which should ideally be deterministic > but it's not because of the following reasons:- > > [https://github.com/apache/kafka/blob/aa4ba8eee8e6f52a9d80a98fb2530b5bcc1b9a11/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L421] > would lead to all configs of type Class to be loaded and this would mean > that they are not loaded in the context of the connectors plugin loader. > IIUC, the current loaded would be the DelegatingClassLoader. This would mean > that the AvroConverter could potentially be loaded from connector2 plugin > path while loading the class. This should be made deterministic as intended. > Also, there could be instances where the Converter itself could potentially > be using ServiceLoader and for that to work when the Converter is created & > configured the current class loader should be set to the one corresponding to > the converter. > > It might also be better to load all the config in
[jira] [Created] (KAFKA-8819) Plugin path for converters not working as intended
Magesh kumar Nandakumar created KAFKA-8819: -- Summary: Plugin path for converters not working as intended Key: KAFKA-8819 URL: https://issues.apache.org/jira/browse/KAFKA-8819 Project: Kafka Issue Type: Bug Components: KafkaConnect Reporter: Magesh kumar Nandakumar Assignee: Magesh kumar Nandakumar KafakConnect allows all plugins to be available via a plugin path mechanism. This allows for classpath isolation. This is not working as designed under the following circumstances for Converters I have 2 directories under plugin path `connector1` and `connector2`. I intend to use AvroConverter and its available in both the plugin directories. Under these circumstances, the Worker attempts to create the Converter available in the plugin director first which should ideally be deterministic but it's not because of the following reasons:- [https://github.com/apache/kafka/blob/aa4ba8eee8e6f52a9d80a98fb2530b5bcc1b9a11/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L421] would lead to all configs of type Class to be loaded and this would mean that they are not loaded in the context of the connectors plugin loader. IIUC, the current loaded would be the DelegatingClassLoader . This would mean that the AvroConverter could potentially be loaded from connector2 plugin path while loading the class. This should be made deterministic as intended. Also, there could be instances where the Converter itself could potentially be using ServiceLoader and for that to work when the Converter is created & configured the current class loader should be set to the one corresponding to the converter. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-4212) Add a key-value store that is a TTL persistent cache
[
https://issues.apache.org/jira/browse/KAFKA-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910715#comment-16910715
]
Sophie Blee-Goldman commented on KAFKA-4212:
[~jamesritt] By the way, you might want to look into FIFO compaction. It's
basically a rocksdb compaction style that deletes the oldest data when the
total size exceeds some threshold. If all you care about is preventing
unbounded growth then this may be even closer to what you want than TTL, and
you should be able to set it using the RocksDBConfigSetter.
> Add a key-value store that is a TTL persistent cache
>
>
> Key: KAFKA-4212
> URL: https://issues.apache.org/jira/browse/KAFKA-4212
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Priority: Major
> Labels: api
>
> Some jobs needs to maintain as state a large set of key-values for some
> period of time. I.e. they need to maintain a TTL cache of values potentially
> larger than memory.
> Currently Kafka Streams provides non-windowed and windowed key-value stores.
> Neither is an exact fit to this use case.
> The {{RocksDBStore}}, a {{KeyValueStore}}, stores one value per key as
> required, but does not support expiration. The TTL option of RocksDB is
> explicitly not used.
> The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment
> dropping, but it stores multiple items per key, based on their timestamp.
> But this store can be repurposed as a cache by fetching the items in reverse
> chronological order and returning the first item found.
> KAFKA-2594 introduced a fixed-capacity in-memory LRU caching store, but here
> we desire a variable-capacity memory-overflowing TTL caching store.
> Although {{RocksDBWindowsStore}} can be repurposed as a cache, it would be
> useful to have an official and proper TTL cache API and implementation.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-8661) Flaky Test RebalanceSourceConnectorsIntegrationTest#testStartTwoConnectors
[
https://issues.apache.org/jira/browse/KAFKA-8661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910713#comment-16910713
]
Randall Hauch commented on KAFKA-8661:
--
Added a PR that hopefully addresses this:
https://github.com/apache/kafka/pull/7224 (see also KAFKA-8391)
> Flaky Test RebalanceSourceConnectorsIntegrationTest#testStartTwoConnectors
> --
>
> Key: KAFKA-8661
> URL: https://issues.apache.org/jira/browse/KAFKA-8661
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, unit tests
>Affects Versions: 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Randall Hauch
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.4.0, 2.3.1
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/224/testReport/junit/org.apache.kafka.connect.integration/RebalanceSourceConnectorsIntegrationTest/testStartTwoConnectors/]
> {quote}java.lang.AssertionError: Condition not met within timeout 3.
> Connector tasks did not start in time. at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:376) at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:353) at
> org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest.testStartTwoConnectors(RebalanceSourceConnectorsIntegrationTest.java:120){quote}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-8391) Flaky Test RebalanceSourceConnectorsIntegrationTest#testDeleteConnector
[
https://issues.apache.org/jira/browse/KAFKA-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910710#comment-16910710
]
ASF GitHub Bot commented on KAFKA-8391:
---
rhauch commented on pull request #7224: KAFKA-8391, KAFKA-8661: Improve flaky
Connect rebalance integration tests
URL: https://github.com/apache/kafka/pull/7224
Increased the timeout by a substantial amount, though this will only take
effect if the test conditions are not met successfully.
Added logic to allow tests to wait for a number of records to have been
written to / consumed from a specific topic. This involved extracting the
previously-duplicated logic in `ConnectorHandle` and TaskHandle (test framework
classes) into a new `RecordLatches` class that can be easily reused by both
those handle classes.
These integration tests are passing were occasionally failing for me
locally, but now I’ve run them successfully nearly a dozen times in a row. If
the Jenkins builds are a bit slower, this might help the tests fail less
frequently.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flaky Test RebalanceSourceConnectorsIntegrationTest#testDeleteConnector
> ---
>
> Key: KAFKA-8391
> URL: https://issues.apache.org/jira/browse/KAFKA-8391
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Randall Hauch
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.1.2, 2.2.2, 2.4.0, 2.3.1
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/4747/testReport/junit/org.apache.kafka.connect.integration/RebalanceSourceConnectorsIntegrationTest/testDeleteConnector/]
> {quote}java.lang.AssertionError: Condition not met within timeout 3.
> Connector tasks did not stop in time. at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:375) at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:352) at
> org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest.testDeleteConnector(RebalanceSourceConnectorsIntegrationTest.java:166){quote}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-8816) RecordCollector offsets updated indirectly by StreamTask
[
https://issues.apache.org/jira/browse/KAFKA-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910707#comment-16910707
]
ASF GitHub Bot commented on KAFKA-8816:
---
cpettitt-confluent commented on pull request #7223: KAFKA-8816: Make offsets
immutable to users of RecordCollector.offsets
URL: https://github.com/apache/kafka/pull/7223
Make offsets immutable to users of RecordCollector.offsets. Fix up an
existing case where offsets could be modified in this way. Add a simple
test to verify offsets cannot be changed externally.
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> RecordCollector offsets updated indirectly by StreamTask
>
>
> Key: KAFKA-8816
> URL: https://issues.apache.org/jira/browse/KAFKA-8816
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Chris Pettitt
>Assignee: Chris Pettitt
>Priority: Major
>
> Currently it is possible to indirectly update the offsets in
> RecordCollectorImpl via the offset read function:
> {code:java}
> @Override
> public Map offsets() {
> return offsets;
> } {code}
> The offsets here is the a private final field in RecordCollectorImpl. It
> appears that the intent is for this field to be updated only when the
> producer acknowledges an offset. However, because it is handed back in a
> mutable form, it is possible to update offsets through this call, as actually
> happens today in StreamTask:
> {code:java}
> protected Map activeTaskCheckpointableOffsets() {
> final Map checkpointableOffsets =
> recordCollector.offsets();
> for (final Map.Entry entry :
> consumedOffsets.entrySet()) {
> checkpointableOffsets.putIfAbsent(entry.getKey(), entry.getValue());
> }
> return checkpointableOffsets;
> }{code}
> Here it is possible to set a new checkpoint if the topic partition is not
> already in the offsets map, which happens for the input topic when we're
> using optimized topologies and a KTable. The effect is that we continue to
> checkpoint the first offset seen (putIfAbsent).
> It seems the correct behavior would be to return a read only view of the
> offsets from RecordCollectorImpl and create a copy of the returned map in
> activeTaskCheckpointableOffsets before we mutate it.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8818) CreatePartitions Request protocol documentation
[
https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fábio Silva updated KAFKA-8818:
---
Description:
CreatePartitions Request protocol documentation contains a invalid type
ARRAY(INT32) (assignment field), it must be INT32.
{code:java}
assignment => ARRAY(INT32){code}
was:CreatePartitions Request protocol documentation contains a invalid type
ARRAY(INT32) (assignment field), it must be INT32.
> CreatePartitions Request protocol documentation
> ---
>
> Key: KAFKA-8818
> URL: https://issues.apache.org/jira/browse/KAFKA-8818
> Project: Kafka
> Issue Type: Bug
> Components: documentation
>Reporter: Fábio Silva
>Priority: Major
> Labels: documentation, protocol-documentation
>
> CreatePartitions Request protocol documentation contains a invalid type
> ARRAY(INT32) (assignment field), it must be INT32.
>
> {code:java}
> assignment => ARRAY(INT32){code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8818) CreatePartitions Request protocol documentation
[
https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fábio Silva updated KAFKA-8818:
---
Priority: Minor (was: Major)
> CreatePartitions Request protocol documentation
> ---
>
> Key: KAFKA-8818
> URL: https://issues.apache.org/jira/browse/KAFKA-8818
> Project: Kafka
> Issue Type: Bug
> Components: documentation
>Reporter: Fábio Silva
>Priority: Minor
> Labels: documentation, protocol-documentation
>
> CreatePartitions Request protocol documentation contains a invalid type
> ARRAY(INT32) (assignment field), it must be INT32.
> Wrong:
> {code:java}
> assignment => ARRAY(INT32){code}
> Correct:
> {code:java}
> assignment => INT32
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8818) CreatePartitions Request protocol documentation
[
https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fábio Silva updated KAFKA-8818:
---
Description:
CreatePartitions Request protocol documentation contains a invalid type
ARRAY(INT32) (assignment field), it must be INT32.
Wrong:
{code:java}
assignment => ARRAY(INT32){code}
Correct:
{code:java}
assignment => INT32
{code}
was:
CreatePartitions Request protocol documentation contains a invalid type
ARRAY(INT32) (assignment field), it must be INT32.
{code:java}
assignment => ARRAY(INT32){code}
> CreatePartitions Request protocol documentation
> ---
>
> Key: KAFKA-8818
> URL: https://issues.apache.org/jira/browse/KAFKA-8818
> Project: Kafka
> Issue Type: Bug
> Components: documentation
>Reporter: Fábio Silva
>Priority: Major
> Labels: documentation, protocol-documentation
>
> CreatePartitions Request protocol documentation contains a invalid type
> ARRAY(INT32) (assignment field), it must be INT32.
> Wrong:
> {code:java}
> assignment => ARRAY(INT32){code}
> Correct:
> {code:java}
> assignment => INT32
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8698) ListOffsets Response protocol documentation
[
https://issues.apache.org/jira/browse/KAFKA-8698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fábio Silva updated KAFKA-8698:
---
Labels: documentation protocol-documentation (was: documentation)
> ListOffsets Response protocol documentation
> ---
>
> Key: KAFKA-8698
> URL: https://issues.apache.org/jira/browse/KAFKA-8698
> Project: Kafka
> Issue Type: Bug
> Components: documentation
>Reporter: Fábio Silva
>Assignee: Asutosh Pandya
>Priority: Minor
> Labels: documentation, protocol-documentation
>
> The documentation of ListOffsets Response (Version: 0) appears to have an
> typo on offsets field name, suffixed with `'`.
> {code:java}
> [offsets']{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8818) CreatePartitions Request protocol documentation
[ https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fábio Silva updated KAFKA-8818: --- Component/s: documentation > CreatePartitions Request protocol documentation > --- > > Key: KAFKA-8818 > URL: https://issues.apache.org/jira/browse/KAFKA-8818 > Project: Kafka > Issue Type: Bug > Components: documentation >Reporter: Fábio Silva >Priority: Major > Labels: documentation, protocol-documentation > > CreatePartitions Request protocol documentation contains a invalid type > ARRAY(INT32) (assignment field), it must be INT32. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8818) CreatePartitions Request protocol documentation
[ https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fábio Silva updated KAFKA-8818: --- Labels: documentation protocol-documentation (was: d) > CreatePartitions Request protocol documentation > --- > > Key: KAFKA-8818 > URL: https://issues.apache.org/jira/browse/KAFKA-8818 > Project: Kafka > Issue Type: Bug >Reporter: Fábio Silva >Priority: Major > Labels: documentation, protocol-documentation > > CreatePartitions Request protocol documentation contains a invalid type > ARRAY(INT32) (assignment field), it must be INT32. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8818) CreatePartitions Request protocol documentation
[ https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fábio Silva updated KAFKA-8818: --- Description: CreatePartitions Request protocol documentation contains a invalid type ARRAY(INT32) (assignment field). It must be INT32. (was: CreatePartitions Request protocol documentation contains a invalid type ARRAY(INT32) (assignment field). It mus be INT32.) > CreatePartitions Request protocol documentation > --- > > Key: KAFKA-8818 > URL: https://issues.apache.org/jira/browse/KAFKA-8818 > Project: Kafka > Issue Type: Bug >Reporter: Fábio Silva >Priority: Major > Labels: d > > CreatePartitions Request protocol documentation contains a invalid type > ARRAY(INT32) (assignment field). It must be INT32. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8818) CreatePartitions Request protocol documentation
[ https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fábio Silva updated KAFKA-8818: --- Description: CreatePartitions Request protocol documentation contains a invalid type ARRAY(INT32) (assignment field), it must be INT32. (was: CreatePartitions Request protocol documentation contains a invalid type ARRAY(INT32) (assignment field). It must be INT32.) > CreatePartitions Request protocol documentation > --- > > Key: KAFKA-8818 > URL: https://issues.apache.org/jira/browse/KAFKA-8818 > Project: Kafka > Issue Type: Bug >Reporter: Fábio Silva >Priority: Major > Labels: d > > CreatePartitions Request protocol documentation contains a invalid type > ARRAY(INT32) (assignment field), it must be INT32. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8818) CreatePartitions Request protocol documentation
[ https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fábio Silva updated KAFKA-8818: --- Labels: d (was: ) > CreatePartitions Request protocol documentation > --- > > Key: KAFKA-8818 > URL: https://issues.apache.org/jira/browse/KAFKA-8818 > Project: Kafka > Issue Type: Bug >Reporter: Fábio Silva >Priority: Major > Labels: d > > CreatePartitions Request protocol documentation contains a invalid type > ARRAY(INT32) (assignment field). It mus be INT32. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KAFKA-8818) CreatePartitions Request protocol documentation
[ https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fábio Silva updated KAFKA-8818: --- Description: CreatePartitions Request protocol documentation contains a invalid type ARRAY(INT32) (assignment field). It mus be INT32. > CreatePartitions Request protocol documentation > --- > > Key: KAFKA-8818 > URL: https://issues.apache.org/jira/browse/KAFKA-8818 > Project: Kafka > Issue Type: Bug >Reporter: Fábio Silva >Priority: Major > > CreatePartitions Request protocol documentation contains a invalid type > ARRAY(INT32) (assignment field). It mus be INT32. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KAFKA-8818) CreatePartitions Request protocol documentation
Fábio Silva created KAFKA-8818: -- Summary: CreatePartitions Request protocol documentation Key: KAFKA-8818 URL: https://issues.apache.org/jira/browse/KAFKA-8818 Project: Kafka Issue Type: Bug Reporter: Fábio Silva -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-4212) Add a key-value store that is a TTL persistent cache
[
https://issues.apache.org/jira/browse/KAFKA-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910650#comment-16910650
]
James Ritt commented on KAFKA-4212:
---
Hi [~guozhang] thanks for input! In the comments above, most of us clamoring
for this feature seem to have usages where there is a need to enable the TTL
just for RocksDB in order to prevent unbounded growth, and that strict
coherency with the topic isn't required. And given the feedback from Sophie &
Matthias, it definitely seemed like we were heading the KIP direction as I
couldn't come up with a good way to signal usage of the TTL RocksDB in the APIs
available now: `RocksDBConfigSetter` can't be repurposed as it currently
exists, we don't want to modify the `Stores` API, and `StreamsConfig`seemed too
high-level. So if anyone has an idea as to how to properly surface this as an
option it would help inform a KPI.
> Add a key-value store that is a TTL persistent cache
>
>
> Key: KAFKA-4212
> URL: https://issues.apache.org/jira/browse/KAFKA-4212
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Priority: Major
> Labels: api
>
> Some jobs needs to maintain as state a large set of key-values for some
> period of time. I.e. they need to maintain a TTL cache of values potentially
> larger than memory.
> Currently Kafka Streams provides non-windowed and windowed key-value stores.
> Neither is an exact fit to this use case.
> The {{RocksDBStore}}, a {{KeyValueStore}}, stores one value per key as
> required, but does not support expiration. The TTL option of RocksDB is
> explicitly not used.
> The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment
> dropping, but it stores multiple items per key, based on their timestamp.
> But this store can be repurposed as a cache by fetching the items in reverse
> chronological order and returning the first item found.
> KAFKA-2594 introduced a fixed-capacity in-memory LRU caching store, but here
> we desire a variable-capacity memory-overflowing TTL caching store.
> Although {{RocksDBWindowsStore}} can be repurposed as a cache, it would be
> useful to have an official and proper TTL cache API and implementation.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-8817) Flaky Test KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest
[
https://issues.apache.org/jira/browse/KAFKA-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Pettitt updated KAFKA-8817:
-
Description:
Error:
{code:java}
org.junit.runners.model.TestTimedOutException: test timed out after 5000
milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1260)
at
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1190)
at
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1167)
at
org.apache.kafka.clients.producer.KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest(KafkaProducerTest.java:894)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
Currently 100% reproducible locally when running the whole test suite. Does not
repro when running this test class individually. This is on the latest upstream
trunk with no changes applied.
was:
Error:
{code:java}
org.junit.runners.model.TestTimedOutException: test timed out after 5000
milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1260)
at
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1190)
at
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1167)
at
org.apache.kafka.clients.producer.KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest(KafkaProducerTest.java:894)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
Currently 100% reproducible locally when running the whole test suite. Does not
repro when running this test class individually.
> Flaky Test KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest
> ---
>
> Key: KAFKA-8817
> URL: https://issues.apache.org/jira/browse/KAFKA-8817
> Project: Kafka
> Issue Type: Bug
> Components: core
>Reporter: Chris Pettitt
>Priority: Major
> Labels: flaky-test
>
> Error:
> {code:java}
> org.junit.runners.model.TestTimedOutException: test timed out after 5000
> milliseconds
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1260)
> at
> org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1190)
> at
> org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1167)
> at
> org.apache.kafka.clients.producer.KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest(KafkaProducerTest.java:894)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
>
[jira] [Created] (KAFKA-8817) Flaky Test KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest
Chris Pettitt created KAFKA-8817:
Summary: Flaky Test
KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest
Key: KAFKA-8817
URL: https://issues.apache.org/jira/browse/KAFKA-8817
Project: Kafka
Issue Type: Bug
Components: core
Reporter: Chris Pettitt
Error:
{code:java}
org.junit.runners.model.TestTimedOutException: test timed out after 5000
milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1260)
at
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1190)
at
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:1167)
at
org.apache.kafka.clients.producer.KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest(KafkaProducerTest.java:894)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
Currently 100% reproducible locally when running the whole test suite. Does not
repro when running this test class individually.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-5140) Flaky ResetIntegrationTest
[
https://issues.apache.org/jira/browse/KAFKA-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-5140:
---
Labels: flaky-test (was: )
> Flaky ResetIntegrationTest
> --
>
> Key: KAFKA-5140
> URL: https://issues.apache.org/jira/browse/KAFKA-5140
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Guozhang Wang
>Priority: Major
> Labels: flaky-test
> Fix For: 0.11.0.2, 1.0.0
>
>
> {noformat}
> org.apache.kafka.streams.integration.ResetIntegrationTest >
> testReprocessingFromScratchAfterResetWithIntermediateUserTopic FAILED
> java.lang.AssertionError:
> Expected: <[KeyValue(2986681642095, 1), KeyValue(2986681642055, 1),
> KeyValue(2986681642075, 1), KeyValue(2986681642035, 1),
> KeyValue(2986681642095, 1), KeyValue(2986681642055, 1),
> KeyValue(2986681642115, 1), KeyValue(2986681642075, 1),
> KeyValue(2986681642075, 2), KeyValue(2986681642095, 2),
> KeyValue(2986681642115, 1), KeyValue(2986681642135, 1),
> KeyValue(2986681642095, 2), KeyValue(2986681642115, 2),
> KeyValue(2986681642155, 1), KeyValue(2986681642135, 1),
> KeyValue(2986681642115, 2), KeyValue(2986681642135, 2),
> KeyValue(2986681642155, 1), KeyValue(2986681642175, 1),
> KeyValue(2986681642135, 2), KeyValue(2986681642155, 2),
> KeyValue(2986681642175, 1), KeyValue(2986681642195, 1),
> KeyValue(2986681642135, 3), KeyValue(2986681642155, 2),
> KeyValue(2986681642175, 2), KeyValue(2986681642195, 1),
> KeyValue(2986681642155, 3), KeyValue(2986681642175, 2),
> KeyValue(2986681642195, 2), KeyValue(2986681642155, 3),
> KeyValue(2986681642175, 3), KeyValue(2986681642195, 2),
> KeyValue(2986681642155, 4), KeyValue(2986681642175, 3),
> KeyValue(2986681642195, 3)]>
> but: was <[KeyValue(2986681642095, 1), KeyValue(2986681642055, 1),
> KeyValue(2986681642075, 1), KeyValue(2986681642035, 1),
> KeyValue(2986681642095, 1), KeyValue(2986681642055, 1),
> KeyValue(2986681642115, 1), KeyValue(2986681642075, 1),
> KeyValue(2986681642075, 2), KeyValue(2986681642095, 2),
> KeyValue(2986681642115, 1), KeyValue(2986681642135, 1),
> KeyValue(2986681642095, 2), KeyValue(2986681642115, 2),
> KeyValue(2986681642155, 1), KeyValue(2986681642135, 1),
> KeyValue(2986681642115, 2), KeyValue(2986681642135, 2),
> KeyValue(2986681642155, 1), KeyValue(2986681642175, 1),
> KeyValue(2986681642135, 2), KeyValue(2986681642155, 2),
> KeyValue(2986681642175, 1), KeyValue(2986681642195, 1),
> KeyValue(2986681642135, 3), KeyValue(2986681642155, 2),
> KeyValue(2986681642175, 2), KeyValue(2986681642195, 1),
> KeyValue(2986681642155, 3), KeyValue(2986681642175, 2),
> KeyValue(2986681642195, 2), KeyValue(2986681642155, 3)]>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> at
> org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterResetWithIntermediateUserTopic(ResetIntegrationTest.java:190)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (KAFKA-8816) RecordCollector offsets updated indirectly by StreamTask
Chris Pettitt created KAFKA-8816:
Summary: RecordCollector offsets updated indirectly by StreamTask
Key: KAFKA-8816
URL: https://issues.apache.org/jira/browse/KAFKA-8816
Project: Kafka
Issue Type: Bug
Components: streams
Reporter: Chris Pettitt
Assignee: Chris Pettitt
Currently it is possible to indirectly update the offsets in
RecordCollectorImpl via the offset read function:
{code:java}
@Override
public Map offsets() {
return offsets;
} {code}
The offsets here is the a private final field in RecordCollectorImpl. It
appears that the intent is for this field to be updated only when the producer
acknowledges an offset. However, because it is handed back in a mutable form,
it is possible to update offsets through this call, as actually happens today
in StreamTask:
{code:java}
protected Map activeTaskCheckpointableOffsets() {
final Map checkpointableOffsets =
recordCollector.offsets();
for (final Map.Entry entry :
consumedOffsets.entrySet()) {
checkpointableOffsets.putIfAbsent(entry.getKey(), entry.getValue());
}
return checkpointableOffsets;
}{code}
Here it is possible to set a new checkpoint if the topic partition is not
already in the offsets map, which happens for the input topic when we're using
optimized topologies and a KTable. The effect is that we continue to checkpoint
the first offset seen (putIfAbsent).
It seems the correct behavior would be to return a read only view of the
offsets from RecordCollectorImpl and create a copy of the returned map in
activeTaskCheckpointableOffsets before we mutate it.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-8562) SASL_SSL still performs reverse DNS lookup despite KAFKA-5051
[
https://issues.apache.org/jira/browse/KAFKA-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910478#comment-16910478
]
Ismael Juma commented on KAFKA-8562:
cc [~rsivaram] [~omkreddy]
> SASL_SSL still performs reverse DNS lookup despite KAFKA-5051
> -
>
> Key: KAFKA-8562
> URL: https://issues.apache.org/jira/browse/KAFKA-8562
> Project: Kafka
> Issue Type: Bug
>Reporter: Badai Aqrandista
>Priority: Minor
>
> When using SASL_SSL, the Kafka client performs a reverse DNS lookup to
> resolve IP to DNS. So, this circumvent the security fix made in KAFKA-5051.
> This is the line of code from AK 2.2 where it performs the lookup:
> https://github.com/apache/kafka/blob/2.2.0/clients/src/main/java/org/apache/kafka/common/network/SaslChannelBuilder.java#L205
> Following log messages show that consumer initially tried to connect with IP
> address 10.0.2.15. Then suddenly it created SaslClient with a hostname:
> {code:java}
> [2019-06-18 06:23:36,486] INFO Kafka commitId: 00d486623990ed9d
> (org.apache.kafka.common.utils.AppInfoParser)
> [2019-06-18 06:23:36,487] DEBUG [Consumer
> clientId=KafkaStore-reader-_schemas, groupId=schema-registry-10.0.2.15-18081]
> Kafka consumer initialized (org.apache.kafka.clients.consumer.KafkaConsumer)
> [2019-06-18 06:23:36,505] DEBUG [Consumer
> clientId=KafkaStore-reader-_schemas, groupId=schema-registry-10.0.2.15-18081]
> Initiating connection to node 10.0.2.15:19094 (id: -1 rack: null) using
> address /10.0.2.15 (org.apache.kafka.clients.NetworkClient)
> [2019-06-18 06:23:36,512] DEBUG Set SASL client state to
> SEND_APIVERSIONS_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2019-06-18 06:23:36,515] DEBUG Creating SaslClient:
> client=null;service=kafka;serviceHostname=quickstart.confluent.io;mechs=[PLAIN]
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> {code}
> Thanks
> Badai
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KAFKA-3410) Unclean leader election and "Halting because log truncation is not allowed"
[ https://issues.apache.org/jira/browse/KAFKA-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910475#comment-16910475 ] Ryne Yang commented on KAFKA-3410: -- experiencing the same issue on kafka 1.1.1. this is causing production broker to crash, and the only way to solve this for us is to delete the follower's partition logs from the logdir and restart the follower. since we don't want to enable unclean leader election. > Unclean leader election and "Halting because log truncation is not allowed" > --- > > Key: KAFKA-3410 > URL: https://issues.apache.org/jira/browse/KAFKA-3410 > Project: Kafka > Issue Type: Bug > Components: replication >Reporter: James Cheng >Priority: Major > Labels: reliability > > I ran into a scenario where one of my brokers would continually shutdown, > with the error message: > [2016-02-25 00:29:39,236] FATAL [ReplicaFetcherThread-0-1], Halting because > log truncation is not allowed for topic test, Current leader 1's latest > offset 0 is less than replica 2's latest offset 151 > (kafka.server.ReplicaFetcherThread) > I managed to reproduce it with the following scenario: > 1. Start broker1, with unclean.leader.election.enable=false > 2. Start broker2, with unclean.leader.election.enable=false > 3. Create topic, single partition, with replication-factor 2. > 4. Write data to the topic. > 5. At this point, both brokers are in the ISR. Broker1 is the partition > leader. > 6. Ctrl-Z on broker2. (Simulates a GC pause or a slow network) Broker2 gets > dropped out of ISR. Broker1 is still the leader. I can still write data to > the partition. > 7. Shutdown Broker1. Hard or controlled, doesn't matter. > 8. rm -rf the log directory of broker1. (This simulates a disk replacement or > full hardware replacement) > 9. Resume broker2. It attempts to connect to broker1, but doesn't succeed > because broker1 is down. At this point, the partition is offline. Can't write > to it. > 10. Resume broker1. Broker1 resumes leadership of the topic. Broker2 attempts > to join ISR, and immediately halts with the error message: > [2016-02-25 00:29:39,236] FATAL [ReplicaFetcherThread-0-1], Halting because > log truncation is not allowed for topic test, Current leader 1's latest > offset 0 is less than replica 2's latest offset 151 > (kafka.server.ReplicaFetcherThread) > I am able to recover by setting unclean.leader.election.enable=true on my > brokers. > I'm trying to understand a couple things: > * In step 10, why is broker1 allowed to resume leadership even though it has > no data? > * In step 10, why is it necessary to stop the entire broker due to one > partition that is in this state? Wouldn't it be possible for the broker to > continue to serve traffic for all the other topics, and just mark this one as > unavailable? > * Would it make sense to allow an operator to manually specify which broker > they want to become the new master? This would give me more control over how > much data loss I am willing to handle. In this case, I would want broker2 to > become the new master. Or, is that possible and I just don't know how to do > it? -- This message was sent by Atlassian Jira (v8.3.2#803003)
