[jira] [Commented] (KAFKA-8818) CreatePartitions Request protocol documentation
[
https://issues.apache.org/jira/browse/KAFKA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070211#comment-17070211
]
ASF GitHub Bot commented on KAFKA-8818:
---
kkonstantine commented on pull request #7250: KAFKA-8818 : Updated
documentation to assign correct datatype.
URL: https://github.com/apache/kafka/pull/7250
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> CreatePartitions Request protocol documentation
> ---
>
> Key: KAFKA-8818
> URL: https://issues.apache.org/jira/browse/KAFKA-8818
> Project: Kafka
> Issue Type: Bug
> Components: documentation
>Reporter: Fábio Silva
>Priority: Major
> Labels: documentation, protocol-documentation
>
> CreatePartitions Request protocol documentation contains a invalid type
> ARRAY(INT32) (assignment field), it must be INT32.
> Wrong:
> {code:java}
> assignment => ARRAY(INT32){code}
> Correct:
> {code:java}
> assignment => INT32
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-7444) expose connector, task IDs to SinkTasks
[ https://issues.apache.org/jira/browse/KAFKA-7444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantine Karantasis updated KAFKA-7444: -- Labels: needs-kip (was: newbie) > expose connector, task IDs to SinkTasks > --- > > Key: KAFKA-7444 > URL: https://issues.apache.org/jira/browse/KAFKA-7444 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Affects Versions: 2.0.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Labels: needs-kip > > SinkTasks don't have access to their ConnectorTaskId, which would be useful > for logging and for using SinkUtils.consumerGroupId(id). > > I suggest an id() method be added to SinkTaskContext and > WorkerSinkTaskContext. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-6566) SourceTask#stop() not called after exception raised in poll()
[
https://issues.apache.org/jira/browse/KAFKA-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070184#comment-17070184
]
ASF GitHub Bot commented on KAFKA-6566:
---
kkonstantine commented on pull request #4577: KAFKA-6566 SourceTask#stop() not
called after exception raised in poll()
URL: https://github.com/apache/kafka/pull/4577
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> SourceTask#stop() not called after exception raised in poll()
> -
>
> Key: KAFKA-6566
> URL: https://issues.apache.org/jira/browse/KAFKA-6566
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 1.0.0
>Reporter: Gunnar Morling
>Assignee: Robert Yokota
>Priority: Blocker
> Fix For: 0.10.2.2, 0.11.0.3, 1.0.2, 1.1.1, 2.0.0
>
>
> Having discussed this with [~rhauch], it has been my assumption that
> {{SourceTask#stop()}} will be called by the Kafka Connect framework in case
> an exception has been raised in {{poll()}}. That's not the case, though. As
> an example see the connector and task below.
> Calling {{stop()}} after an exception in {{poll()}} seems like a very useful
> action to take, as it'll allow the task to clean up any resources such as
> releasing any database connections, right after that failure and not only
> once the connector is stopped.
> {code}
> package com.example;
> import java.util.Collections;
> import java.util.List;
> import java.util.Map;
> import org.apache.kafka.common.config.ConfigDef;
> import org.apache.kafka.connect.connector.Task;
> import org.apache.kafka.connect.source.SourceConnector;
> import org.apache.kafka.connect.source.SourceRecord;
> import org.apache.kafka.connect.source.SourceTask;
> public class TestConnector extends SourceConnector {
> @Override
> public String version() {
> return null;
> }
> @Override
> public void start(Map props) {
> }
> @Override
> public Class taskClass() {
> return TestTask.class;
> }
> @Override
> public List> taskConfigs(int maxTasks) {
> return Collections.singletonList(Collections.singletonMap("foo",
> "bar"));
> }
> @Override
> public void stop() {
> }
> @Override
> public ConfigDef config() {
> return new ConfigDef();
> }
> public static class TestTask extends SourceTask {
> @Override
> public String version() {
> return null;
> }
> @Override
> public void start(Map props) {
> }
> @Override
> public List poll() throws InterruptedException {
> throw new RuntimeException();
> }
> @Override
> public void stop() {
> System.out.println("stop() called");
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9784) Add OffsetFetch to the concurrent coordinator test
[ https://issues.apache.org/jira/browse/KAFKA-9784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070177#comment-17070177 ] ASF GitHub Bot commented on KAFKA-9784: --- abbccdda commented on pull request #8383: KAFKA-9784: Add OffsetFetch to group concurrency test URL: https://github.com/apache/kafka/pull/8383 As title suggested, consumers would first do an OffsetFetch before starting the normal processing. It makes sense to add it to the concurrent test suite to verify whether there would be a blocking behavior. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add OffsetFetch to the concurrent coordinator test > -- > > Key: KAFKA-9784 > URL: https://issues.apache.org/jira/browse/KAFKA-9784 > Project: Kafka > Issue Type: Improvement >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Major > > Normally consumers would first do an OffsetFetch before starting the normal > processing. It makes sense to add it to the concurrent test suite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9784) Add OffsetFetch to the concurrent coordinator test
Boyang Chen created KAFKA-9784: -- Summary: Add OffsetFetch to the concurrent coordinator test Key: KAFKA-9784 URL: https://issues.apache.org/jira/browse/KAFKA-9784 Project: Kafka Issue Type: Improvement Reporter: Boyang Chen Assignee: Boyang Chen Normally consumers would first do an OffsetFetch before starting the normal processing. It makes sense to add it to the concurrent test suite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-5538) User-specified Producer/Consumer config doesn't effect with KafkaBackingStore(Config/Status/Offset)
[ https://issues.apache.org/jira/browse/KAFKA-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070171#comment-17070171 ] ASF GitHub Bot commented on KAFKA-5538: --- kkonstantine commented on pull request #3458: KAFKA-5538: User-specified Producer/Consumer config doesn't effect wi… URL: https://github.com/apache/kafka/pull/3458 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > User-specified Producer/Consumer config doesn't effect with > KafkaBackingStore(Config/Status/Offset) > --- > > Key: KAFKA-5538 > URL: https://issues.apache.org/jira/browse/KAFKA-5538 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 0.11.0.0 >Reporter: Aegeaner >Assignee: Aegeaner >Priority: Major > > For configuration of Kafka source and Kafka sink tasks, the same parameters > can be used but need to be prefixed with consumer. and producer. The worker > will take off the prefix and get user specified configurations, but the > KafkaBackingStores will not. All the three KafkaBackingStores just took > originals from the Kafka config without taking off the prefix, so the > producer/consumer will ignore these configurations. (e.g. Kerberos configs) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-5540) Deprecate and remove internal converter configs
[ https://issues.apache.org/jira/browse/KAFKA-5540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070165#comment-17070165 ] ASF GitHub Bot commented on KAFKA-5540: --- kkonstantine commented on pull request #4467: KAFKA-5540 : Deprecate and remove internal converter configs URL: https://github.com/apache/kafka/pull/4467 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Deprecate and remove internal converter configs > --- > > Key: KAFKA-5540 > URL: https://issues.apache.org/jira/browse/KAFKA-5540 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 0.11.0.0 >Reporter: Ewen Cheslack-Postava >Assignee: Chris Egerton >Priority: Major > Labels: needs-kip > Fix For: 2.0.0 > > > The internal.key.converter and internal.value.converter were original exposed > as configs because a) they are actually pluggable and b) providing a default > would require relying on the JsonConverter always being available, which > until we had classloader isolation it was possible might be removed for > compatibility reasons. > However, this has ultimately just caused a lot more trouble and confusion > than it is worth. We should deprecate the configs, give them a default of > JsonConverter (which is also kind of nice since it results in human-readable > data in the internal topics), and then ultimately remove them in the next > major version. > These are all public APIs so this will need a small KIP before we can make > the change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9780) Deprecate commit records without record metadata
[
https://issues.apache.org/jira/browse/KAFKA-9780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070159#comment-17070159
]
Mario Molina commented on KAFKA-9780:
-
Pull request:
[https://github.com/apache/kafka/pull/8379]
> Deprecate commit records without record metadata
>
>
> Key: KAFKA-9780
> URL: https://issues.apache.org/jira/browse/KAFKA-9780
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 2.4.1
>Reporter: Mario Molina
>Assignee: Mario Molina
>Priority: Minor
> Fix For: 2.5.0, 2.6.0
>
>
> Since KIP-382 (MirrorMaker 2.0) a new method {{commitRecord}} was included in
> {{SourceTask}} class to be called by the worker adding a new parameter with
> the record metadata. The old {{commitRecord}} method is called and from the
> new one and it's preserved just for backwards compatibility.
> The idea is to deprecate this method so that we could remove it in a future
> release.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9685) Solve Set concatenation perf issue in AclAuthorizer
[ https://issues.apache.org/jira/browse/KAFKA-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070153#comment-17070153 ] ASF GitHub Bot commented on KAFKA-9685: --- lbradstreet commented on pull request #8382: KAFKA-9685: PT2, avoid unnecessary set creation in ACL matching URL: https://github.com/apache/kafka/pull/8382 https://github.com/apache/kafka/pull/8261 went a long way to solving some of the ACL performance issues. I don't think we need to create sets at all for the `find` and `isEmpty` calls. ``` Before: Benchmark(aclCount) (resourceCount) Mode Cnt ScoreError Units AclAuthorizerBenchmark.testAclsIterator 5 5000 avgt 15 0.430 ± 0.004 ms/op AclAuthorizerBenchmark.testAclsIterator 51 avgt 15 0.980 ± 0.007 ms/op AclAuthorizerBenchmark.testAclsIterator 55 avgt 15 11.191 ± 0.032 ms/op AclAuthorizerBenchmark.testAclsIterator 10 5000 avgt 15 0.880 ± 0.007 ms/op AclAuthorizerBenchmark.testAclsIterator 101 avgt 15 2.642 ± 0.029 ms/op AclAuthorizerBenchmark.testAclsIterator 105 avgt 15 26.361 ± 0.242 ms/op AclAuthorizerBenchmark.testAclsIterator 15 5000 avgt 15 1.655 ± 0.024 ms/op AclAuthorizerBenchmark.testAclsIterator 151 avgt 15 5.276 ± 0.041 ms/op AclAuthorizerBenchmark.testAclsIterator 155 avgt 15 40.702 ± 0.574 ms/op AclAuthorizerBenchmark.testAuthorizer 5 5000 avgt 15 0.202 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer 51 avgt 15 0.233 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer 55 avgt 15 0.424 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer10 5000 avgt 15 0.202 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer101 avgt 15 0.253 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer105 avgt 15 0.423 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer15 5000 avgt 15 0.198 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer151 avgt 15 0.242 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer155 avgt 15 0.391 ± 0.002 ms/op JMH benchmarks done After: Benchmark(aclCount) (resourceCount) Mode Cnt ScoreError Units AclAuthorizerBenchmark.testAclsIterator 5 5000 avgt 15 0.504 ± 0.164 ms/op AclAuthorizerBenchmark.testAclsIterator 51 avgt 15 1.038 ± 0.271 ms/op AclAuthorizerBenchmark.testAclsIterator 55 avgt 15 11.767 ± 0.028 ms/op AclAuthorizerBenchmark.testAclsIterator 10 5000 avgt 15 0.827 ± 0.016 ms/op AclAuthorizerBenchmark.testAclsIterator 101 avgt 15 2.801 ± 0.027 ms/op AclAuthorizerBenchmark.testAclsIterator 105 avgt 15 26.157 ± 0.191 ms/op AclAuthorizerBenchmark.testAclsIterator 15 5000 avgt 15 1.814 ± 0.053 ms/op AclAuthorizerBenchmark.testAclsIterator 151 avgt 15 5.420 ± 0.065 ms/op AclAuthorizerBenchmark.testAclsIterator 155 avgt 15 41.372 ± 0.659 ms/op AclAuthorizerBenchmark.testAuthorizer 5 5000 avgt 15 0.064 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer 51 avgt 15 0.070 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer 55 avgt 15 0.240 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer10 5000 avgt 15 0.055 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer101 avgt 15 0.084 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer105 avgt 15 0.249 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer15 5000 avgt 15 0.057 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer151 avgt 15 0.084 ± 0.001 ms/op AclAuthorizerBenchmark.testAuthorizer155 avgt 15 0.243 ± 0.001 ms/op ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For
[jira] [Commented] (KAFKA-3333) Alternative Partitioner to Support "Always Round-Robin" partitioning
[ https://issues.apache.org/jira/browse/KAFKA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070148#comment-17070148 ] ASF GitHub Bot commented on KAFKA-: --- kkonstantine commented on pull request #1012: [KAFKA-] - Add RoundRobinPartitioner URL: https://github.com/apache/kafka/pull/1012 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Alternative Partitioner to Support "Always Round-Robin" partitioning > > > Key: KAFKA- > URL: https://issues.apache.org/jira/browse/KAFKA- > Project: Kafka > Issue Type: New Feature > Components: clients >Reporter: Stephen Powis >Assignee: M. Manna >Priority: Major > Labels: kip > Fix For: 2.4.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > KIP: > [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89070828] > Please Look into KAFKA-7358 for the official description ** > The > [DefaultPartitioner|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java] > typically distributes using the hash of the keybytes, and falls back to > round robin if there is no key. But there is currently no way to do Round > Robin partitioning if you have keys on your messages without writing your own > partitioning implementation. > I think it'd be helpful to have an implementation of straight Round Robin > partitioning included with the library. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-3720) Remove BufferExhaustException from doSend() in KafkaProducer
[ https://issues.apache.org/jira/browse/KAFKA-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070051#comment-17070051 ] Sönke Liebau commented on KAFKA-3720: - Is anybody still working on this? > Remove BufferExhaustException from doSend() in KafkaProducer > > > Key: KAFKA-3720 > URL: https://issues.apache.org/jira/browse/KAFKA-3720 > Project: Kafka > Issue Type: Bug >Reporter: Mayuresh Gharat >Assignee: Mayuresh Gharat >Priority: Major > > KafkaProducer no longer throws BufferExhaustException. We should remove it > from the catch clause. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (KAFKA-9713) Remove BufferExhausedException
[
https://issues.apache.org/jira/browse/KAFKA-9713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sönke Liebau updated KAFKA-9713:
Comment: was deleted
(was: Is anybody still working on this? )
> Remove BufferExhausedException
> --
>
> Key: KAFKA-9713
> URL: https://issues.apache.org/jira/browse/KAFKA-9713
> Project: Kafka
> Issue Type: Task
> Components: producer
>Reporter: Brian Byrne
>Priority: Trivial
>
> BufferExhaustedException was deprecated in 0.9.0.0, and the corresponding
> block.on.buffer.full property has since been removed. The exception should
> follow.
> {quote}Deprecations in 0.9.0.0
> The producer config block.on.buffer.full has been deprecated and will be
> removed in future release. Currently its default value has been changed to
> false. The KafkaProducer will no longer throw BufferExhaustedException but
> instead will use max.block.ms value to block, after which it will throw a
> TimeoutException. If block.on.buffer.full property is set to true explicitly,
> it will set the max.block.ms to Long.MAX_VALUE and metadata.fetch.timeout.ms
> will not be honoured{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9713) Remove BufferExhausedException
[
https://issues.apache.org/jira/browse/KAFKA-9713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070049#comment-17070049
]
Sönke Liebau commented on KAFKA-9713:
-
Is anybody still working on this?
> Remove BufferExhausedException
> --
>
> Key: KAFKA-9713
> URL: https://issues.apache.org/jira/browse/KAFKA-9713
> Project: Kafka
> Issue Type: Task
> Components: producer
>Reporter: Brian Byrne
>Priority: Trivial
>
> BufferExhaustedException was deprecated in 0.9.0.0, and the corresponding
> block.on.buffer.full property has since been removed. The exception should
> follow.
> {quote}Deprecations in 0.9.0.0
> The producer config block.on.buffer.full has been deprecated and will be
> removed in future release. Currently its default value has been changed to
> false. The KafkaProducer will no longer throw BufferExhaustedException but
> instead will use max.block.ms value to block, after which it will throw a
> TimeoutException. If block.on.buffer.full property is set to true explicitly,
> it will set the max.block.ms to Long.MAX_VALUE and metadata.fetch.timeout.ms
> will not be honoured{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions
[
https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070032#comment-17070032
]
Evan Williams commented on KAFKA-4084:
--
[~sql_consulting]
No pressure from my side, but how's things with the patch going ? :)
> automated leader rebalance causes replication downtime for clusters with too
> many partitions
>
>
> Key: KAFKA-4084
> URL: https://issues.apache.org/jira/browse/KAFKA-4084
> Project: Kafka
> Issue Type: Bug
> Components: controller
>Affects Versions: 0.8.2.2, 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Tom Crayford
>Priority: Major
> Labels: reliability
> Fix For: 1.1.0
>
>
> If you enable {{auto.leader.rebalance.enable}} (which is on by default), and
> you have a cluster with many partitions, there is a severe amount of
> replication downtime following a restart. This causes
> `UnderReplicatedPartitions` to fire, and replication is paused.
> This is because the current automated leader rebalance mechanism changes
> leaders for *all* imbalanced partitions at once, instead of doing it
> gradually. This effectively stops all replica fetchers in the cluster
> (assuming there are enough imbalanced partitions), and restarts them. This
> can take minutes on busy clusters, during which no replication is happening
> and user data is at risk. Clients with {{acks=-1}} also see issues at this
> time, because replication is effectively stalled.
> To quote Todd Palino from the mailing list:
> bq. There is an admin CLI command to trigger the preferred replica election
> manually. There is also a broker configuration “auto.leader.rebalance.enable”
> which you can set to have the broker automatically perform the PLE when
> needed. DO NOT USE THIS OPTION. There are serious performance issues when
> doing so, especially on larger clusters. It needs some development work that
> has not been fully identified yet.
> This setting is extremely useful for smaller clusters, but with high
> partition counts causes the huge issues stated above.
> One potential fix could be adding a new configuration for the number of
> partitions to do automated leader rebalancing for at once, and *stop* once
> that number of leader rebalances are in flight, until they're done. There may
> be better mechanisms, and I'd love to hear if anybody has any ideas.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9783) Flaky Test QueryableStateIntegrationTest#concurrentAccesses
[
https://issues.apache.org/jira/browse/KAFKA-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069981#comment-17069981
]
Matthias J. Sax commented on KAFKA-9783:
And again:
[https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1461/testReport/org.apache.kafka.streams.integration/QueryableStateIntegrationTest/concurrentAccesses/]
[https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/5481/consoleFull]
> Flaky Test QueryableStateIntegrationTest#concurrentAccesses
> ---
>
> Key: KAFKA-9783
> URL: https://issues.apache.org/jira/browse/KAFKA-9783
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.6.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1464/consoleFull]
> {quote}*02:17:54*
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest >
> concurrentAccesses FAILED*02:17:54* java.lang.AssertionError: Did not
> receive all 48 records from topic output-concurrent-2 within 12
> ms*02:17:54* Expected: is a value equal to or greater than <48>*02:17:54*
> but: <0> was less than <48>*02:17:54* at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)*02:17:54*
> at
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinValuesRecordsReceived$6(IntegrationTestUtils.java:691)*02:17:54*
> at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:415)*02:17:54*
> at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:383)*02:17:54*
> at
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinValuesRecordsReceived(IntegrationTestUtils.java:687)*02:17:54*
> at
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.waitUntilAtLeastNumRecordProcessed(QueryableStateIntegrationTest.java:1199)*02:17:54*
> at
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:649){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9783) Flaky Test QueryableStateIntegrationTest#concurrentAccesses
[
https://issues.apache.org/jira/browse/KAFKA-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069979#comment-17069979
]
Matthias J. Sax commented on KAFKA-9783:
[https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1463/testReport/junit/org.apache.kafka.streams.integration/QueryableStateIntegrationTest/concurrentAccesses/]
and
[https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/5483/consoleFull]
> Flaky Test QueryableStateIntegrationTest#concurrentAccesses
> ---
>
> Key: KAFKA-9783
> URL: https://issues.apache.org/jira/browse/KAFKA-9783
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.6.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1464/consoleFull]
> {quote}*02:17:54*
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest >
> concurrentAccesses FAILED*02:17:54* java.lang.AssertionError: Did not
> receive all 48 records from topic output-concurrent-2 within 12
> ms*02:17:54* Expected: is a value equal to or greater than <48>*02:17:54*
> but: <0> was less than <48>*02:17:54* at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)*02:17:54*
> at
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinValuesRecordsReceived$6(IntegrationTestUtils.java:691)*02:17:54*
> at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:415)*02:17:54*
> at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:383)*02:17:54*
> at
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinValuesRecordsReceived(IntegrationTestUtils.java:687)*02:17:54*
> at
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.waitUntilAtLeastNumRecordProcessed(QueryableStateIntegrationTest.java:1199)*02:17:54*
> at
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:649){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9600) EndTxn handler should check strict epoch equality
[ https://issues.apache.org/jira/browse/KAFKA-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069978#comment-17069978 ] ASF GitHub Bot commented on KAFKA-9600: --- hachikuji commented on pull request #8164: KAFKA-9600: EndTxn should enforce strict epoch checking if from client URL: https://github.com/apache/kafka/pull/8164 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > EndTxn handler should check strict epoch equality > - > > Key: KAFKA-9600 > URL: https://issues.apache.org/jira/browse/KAFKA-9600 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Assignee: Boyang Chen >Priority: Major > > The EndTxn path in TransactionCoordinator is shared between direct calls to > EndTxn from the client and internal transaction abort logic. To support the > latter, the code is written to allow an epoch bump. However, if the client > bumps the epoch unexpectedly (e.g. due to a buggy implementation), then the > internal invariants are violated which results in a hanging transaction. > Specifically, the transaction is left in a pending state because the epoch > following append to the log does not match what we expect. > To fix this, we should ensure that an EndTxn from the client checks for > strict epoch equality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9600) EndTxn handler should check strict epoch equality
[ https://issues.apache.org/jira/browse/KAFKA-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-9600. Fix Version/s: 2.6.0 Resolution: Fixed > EndTxn handler should check strict epoch equality > - > > Key: KAFKA-9600 > URL: https://issues.apache.org/jira/browse/KAFKA-9600 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Assignee: Boyang Chen >Priority: Major > Fix For: 2.6.0 > > > The EndTxn path in TransactionCoordinator is shared between direct calls to > EndTxn from the client and internal transaction abort logic. To support the > latter, the code is written to allow an epoch bump. However, if the client > bumps the epoch unexpectedly (e.g. due to a buggy implementation), then the > internal invariants are violated which results in a hanging transaction. > Specifically, the transaction is left in a pending state because the epoch > following append to the log does not match what we expect. > To fix this, we should ensure that an EndTxn from the client checks for > strict epoch equality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9783) Flaky Test QueryableStateIntegrationTest#concurrentAccesses
[
https://issues.apache.org/jira/browse/KAFKA-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069977#comment-17069977
]
Matthias J. Sax commented on KAFKA-9783:
Also: [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/5482/console]
> Flaky Test QueryableStateIntegrationTest#concurrentAccesses
> ---
>
> Key: KAFKA-9783
> URL: https://issues.apache.org/jira/browse/KAFKA-9783
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.6.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1464/consoleFull]
> {quote}*02:17:54*
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest >
> concurrentAccesses FAILED*02:17:54* java.lang.AssertionError: Did not
> receive all 48 records from topic output-concurrent-2 within 12
> ms*02:17:54* Expected: is a value equal to or greater than <48>*02:17:54*
> but: <0> was less than <48>*02:17:54* at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)*02:17:54*
> at
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinValuesRecordsReceived$6(IntegrationTestUtils.java:691)*02:17:54*
> at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:415)*02:17:54*
> at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:383)*02:17:54*
> at
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinValuesRecordsReceived(IntegrationTestUtils.java:687)*02:17:54*
> at
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.waitUntilAtLeastNumRecordProcessed(QueryableStateIntegrationTest.java:1199)*02:17:54*
> at
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:649){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9783) Flaky Test QueryableStateIntegrationTest#concurrentAccesses
[
https://issues.apache.org/jira/browse/KAFKA-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069973#comment-17069973
]
Matthias J. Sax commented on KAFKA-9783:
[https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1462/testReport/junit/org.apache.kafka.streams.integration/QueryableStateIntegrationTest/concurrentAccesses/]
and
[https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1462/testReport/junit/org.apache.kafka.streams.integration/QueryableStateIntegrationTest/concurrentAccesses_2/]
with different error:
{quote}java.nio.file.DirectoryNotEmptyException:
/tmp/state-queryable-state-137107627802392495271/queryable-state-13/1_0 at
sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126) at
org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:802) at
org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:772) at
java.nio.file.Files.walkFileTree(Files.java:2688) at
java.nio.file.Files.walkFileTree(Files.java:2742) at
org.apache.kafka.common.utils.Utils.delete(Utils.java:772) at
org.apache.kafka.common.utils.Utils.delete(Utils.java:758) at
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.purgeLocalStreamsState(IntegrationTestUtils.java:125)
at
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.shutdown(QueryableStateIntegrationTest.java:228){quote}
> Flaky Test QueryableStateIntegrationTest#concurrentAccesses
> ---
>
> Key: KAFKA-9783
> URL: https://issues.apache.org/jira/browse/KAFKA-9783
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.6.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1464/consoleFull]
> {quote}*02:17:54*
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest >
> concurrentAccesses FAILED*02:17:54* java.lang.AssertionError: Did not
> receive all 48 records from topic output-concurrent-2 within 12
> ms*02:17:54* Expected: is a value equal to or greater than <48>*02:17:54*
> but: <0> was less than <48>*02:17:54* at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)*02:17:54*
> at
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinValuesRecordsReceived$6(IntegrationTestUtils.java:691)*02:17:54*
> at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:415)*02:17:54*
> at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:383)*02:17:54*
> at
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinValuesRecordsReceived(IntegrationTestUtils.java:687)*02:17:54*
> at
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.waitUntilAtLeastNumRecordProcessed(QueryableStateIntegrationTest.java:1199)*02:17:54*
> at
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:649){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-7965) Flaky Test ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
[
https://issues.apache.org/jira/browse/KAFKA-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069971#comment-17069971
]
Matthias J. Sax commented on KAFKA-7965:
[https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/5484/consoleFull]
> Flaky Test
> ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
>
>
> Key: KAFKA-7965
> URL: https://issues.apache.org/jira/browse/KAFKA-7965
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 1.1.1, 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/21/]
> {quote}java.lang.AssertionError: Received 0, expected at least 68 at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.ConsumerBounceTest.receiveAndCommit(ConsumerBounceTest.scala:557)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1(ConsumerBounceTest.scala:320)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1$adapted(ConsumerBounceTest.scala:319)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup(ConsumerBounceTest.scala:319){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9783) Flaky Test QueryableStateIntegrationTest#concurrentAccesses
Matthias J. Sax created KAFKA-9783:
--
Summary: Flaky Test
QueryableStateIntegrationTest#concurrentAccesses
Key: KAFKA-9783
URL: https://issues.apache.org/jira/browse/KAFKA-9783
Project: Kafka
Issue Type: Bug
Components: streams, unit tests
Affects Versions: 2.6.0
Reporter: Matthias J. Sax
[https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1464/consoleFull]
{quote}*02:17:54*
org.apache.kafka.streams.integration.QueryableStateIntegrationTest >
concurrentAccesses FAILED*02:17:54* java.lang.AssertionError: Did not
receive all 48 records from topic output-concurrent-2 within 12
ms*02:17:54* Expected: is a value equal to or greater than <48>*02:17:54*
but: <0> was less than <48>*02:17:54* at
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)*02:17:54*
at
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinValuesRecordsReceived$6(IntegrationTestUtils.java:691)*02:17:54*
at
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:415)*02:17:54*
at
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:383)*02:17:54*
at
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinValuesRecordsReceived(IntegrationTestUtils.java:687)*02:17:54*
at
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.waitUntilAtLeastNumRecordProcessed(QueryableStateIntegrationTest.java:1199)*02:17:54*
at
org.apache.kafka.streams.integration.QueryableStateIntegrationTest.concurrentAccesses(QueryableStateIntegrationTest.java:649){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9166) Implement MetadataFetch API
[ https://issues.apache.org/jira/browse/KAFKA-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069351#comment-17069351 ] jacky edited comment on KAFKA-9166 at 3/28/20, 3:30 PM: it is a patch available? In the first version, we can use the zxid of the zookeeper, because it increases globally. was (Author: jacky0123): it is a patch available? In the first version, we can use the zxid of the zookeeper, because it increase globally. > Implement MetadataFetch API > --- > > Key: KAFKA-9166 > URL: https://issues.apache.org/jira/browse/KAFKA-9166 > Project: Kafka > Issue Type: Sub-task > Components: controller >Reporter: Viktor Somogyi-Vass >Assignee: Colin McCabe >Priority: Major > > Brief description of the ask is mentioned in KIP-500's > [BrokerMetadataManagement|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-BrokerMetadataManagement] > Instead of the controller pushing out updates to the other brokers, those > brokers will fetch updates from the active controller via the new > MetadataFetch API. > A MetadataFetch is similar to a fetch request. Just like with a fetch > request, the broker will track the offset of the last updates it fetched, and > only request newer updates from the active controller. > The broker will persist the metadata it fetched to disk. This will allow the > broker to start up very quickly, even if there are hundreds of thousands or > even millions of partitions. (Note that since this persistence is an > optimization, we can leave it out of the first version, if it makes > development easier.) > Most of the time, the broker should only need to fetch the deltas, not the > full state. However, if the broker is too far behind the active controller, > or if the broker has no cached metadata at all, the controller will send a > full metadata image rather than a series of deltas. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9166) Implement MetadataFetch API
[ https://issues.apache.org/jira/browse/KAFKA-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069351#comment-17069351 ] jacky edited comment on KAFKA-9166 at 3/28/20, 3:21 PM: it is a patch available? In the first version, we can use the zxid of the zookeeper, because it increase globally. was (Author: jacky0123): it is a patch available? > Implement MetadataFetch API > --- > > Key: KAFKA-9166 > URL: https://issues.apache.org/jira/browse/KAFKA-9166 > Project: Kafka > Issue Type: Sub-task > Components: controller >Reporter: Viktor Somogyi-Vass >Assignee: Colin McCabe >Priority: Major > > Brief description of the ask is mentioned in KIP-500's > [BrokerMetadataManagement|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-BrokerMetadataManagement] > Instead of the controller pushing out updates to the other brokers, those > brokers will fetch updates from the active controller via the new > MetadataFetch API. > A MetadataFetch is similar to a fetch request. Just like with a fetch > request, the broker will track the offset of the last updates it fetched, and > only request newer updates from the active controller. > The broker will persist the metadata it fetched to disk. This will allow the > broker to start up very quickly, even if there are hundreds of thousands or > even millions of partitions. (Note that since this persistence is an > optimization, we can leave it out of the first version, if it makes > development easier.) > Most of the time, the broker should only need to fetch the deltas, not the > full state. However, if the broker is too far behind the active controller, > or if the broker has no cached metadata at all, the controller will send a > full metadata image rather than a series of deltas. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9713) Remove BufferExhausedException
[
https://issues.apache.org/jira/browse/KAFKA-9713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069457#comment-17069457
]
Sönke Liebau commented on KAFKA-9713:
-
As [~ijuma] said in the comments to the pull request, there is already an issue
that predates this one (KAFKA-3720) that has seen a few discussions around
this. I think that issue should be used for further discussions.
Also, BufferExhaustedException is part of the public API and has not been
deprecated yet as far as I can tell, so should not simply be removed.
> Remove BufferExhausedException
> --
>
> Key: KAFKA-9713
> URL: https://issues.apache.org/jira/browse/KAFKA-9713
> Project: Kafka
> Issue Type: Task
> Components: producer
>Reporter: Brian Byrne
>Priority: Trivial
>
> BufferExhaustedException was deprecated in 0.9.0.0, and the corresponding
> block.on.buffer.full property has since been removed. The exception should
> follow.
> {quote}Deprecations in 0.9.0.0
> The producer config block.on.buffer.full has been deprecated and will be
> removed in future release. Currently its default value has been changed to
> false. The KafkaProducer will no longer throw BufferExhaustedException but
> instead will use max.block.ms value to block, after which it will throw a
> TimeoutException. If block.on.buffer.full property is set to true explicitly,
> it will set the max.block.ms to Long.MAX_VALUE and metadata.fetch.timeout.ms
> will not be honoured{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9782) Kafka Connect InsertField transform - Add the ability to insert event's Key into Value
Ryan Tomczik created KAFKA-9782: --- Summary: Kafka Connect InsertField transform - Add the ability to insert event's Key into Value Key: KAFKA-9782 URL: https://issues.apache.org/jira/browse/KAFKA-9782 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Ryan Tomczik I'm using Debezium to pull change data capture events from a Mongo DB and write them to S3 with the Confluent S3 Sink. The problem is Debezium stores the document's key in each event's key and the S3 connector discards this key. I need the ability to insert the key as a new field in the event value. It seems that this would fit in perfectly into the InsertField transform or create a new transform KeyToValue. Here is an example of someone else running into this same limitation and creating a custom transform. [https://gist.github.com/shashidesai/aaf72489165c6a0fd73a3b51e5a8892a] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9166) Implement MetadataFetch API
[ https://issues.apache.org/jira/browse/KAFKA-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069351#comment-17069351 ] jacky commented on KAFKA-9166: -- it is a patch available? > Implement MetadataFetch API > --- > > Key: KAFKA-9166 > URL: https://issues.apache.org/jira/browse/KAFKA-9166 > Project: Kafka > Issue Type: Sub-task > Components: controller >Reporter: Viktor Somogyi-Vass >Assignee: Colin McCabe >Priority: Major > > Brief description of the ask is mentioned in KIP-500's > [BrokerMetadataManagement|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-BrokerMetadataManagement] > Instead of the controller pushing out updates to the other brokers, those > brokers will fetch updates from the active controller via the new > MetadataFetch API. > A MetadataFetch is similar to a fetch request. Just like with a fetch > request, the broker will track the offset of the last updates it fetched, and > only request newer updates from the active controller. > The broker will persist the metadata it fetched to disk. This will allow the > broker to start up very quickly, even if there are hundreds of thousands or > even millions of partitions. (Note that since this persistence is an > optimization, we can leave it out of the first version, if it makes > development easier.) > Most of the time, the broker should only need to fetch the deltas, not the > full state. However, if the broker is too far behind the active controller, > or if the broker has no cached metadata at all, the controller will send a > full metadata image rather than a series of deltas. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9781) TimestampConverter / Allow to specify a time zone when converting unix epoch to string
Al created KAFKA-9781: - Summary: TimestampConverter / Allow to specify a time zone when converting unix epoch to string Key: KAFKA-9781 URL: https://issues.apache.org/jira/browse/KAFKA-9781 Project: Kafka Issue Type: Wish Components: KafkaConnect Reporter: Al TimestampConverter can convert a unix epoch value (long; number of milliseconds since Jan 01 1970 00:00 GMT) to string. However, when doing such conversion, the string result depends on the time zone used. TimestampConverter uses UTC (i.e. GMT) for the conversion and does not allow to change it. But I would need this in order to get the date/time representation in my local time zone. So I propose to introduce another config parameter (optional) for "target.type=string": *timeZone* (use java name for that). If no time zone is specified, UTC should be used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-4587) Rethink Unification of Caching with Dedupping
[
https://issues.apache.org/jira/browse/KAFKA-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069296#comment-17069296
]
Matthias J. Sax commented on KAFKA-4587:
It is still relevant. In 2.1 release we added `suppress()` operator (cf
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables])
that is one step into this direction but caching and forwarding is still
coupled in the DSL.
Overall, for cleaner semantics it would be good to decouple caching for local
stores (and writing into the changelog topics) from downstream rate control as
we have `suppress()` for downstream rate control now.
> Rethink Unification of Caching with Dedupping
> -
>
> Key: KAFKA-4587
> URL: https://issues.apache.org/jira/browse/KAFKA-4587
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Priority: Major
>
> This is discussed in PR https://github.com/apache/kafka/pull/1588
> In order to support user-customizable state store suppliers in the DSL we did
> the following:
> 1) Introduce a {{TupleForwarder}} to forward tuples from cached stores that
> is wrapping user customized stores.
> 2) Narrow the scope to only dedup on forwarding if it is the default
> CachingXXStore with wrapper RocksDB.
> With this, the unification of dedupping and caching is less useful now, and
> we are complicating the inner implementations for forwarding a lot. We need
> to re-consider this feature with finer granularity of turning on / off
> caching per store, potentially with explicit triggers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-6443) KTable involved in multiple joins could result in duplicate results
[
https://issues.apache.org/jira/browse/KAFKA-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069292#comment-17069292
]
Matthias J. Sax commented on KAFKA-6443:
Nobody is working on this atm – but we take PRs :)
However, there is related work: KAFKA-8770 that may land in 2.6 release.
> KTable involved in multiple joins could result in duplicate results
> ---
>
> Key: KAFKA-6443
> URL: https://issues.apache.org/jira/browse/KAFKA-6443
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Priority: Major
>
> Consider the following multi table-table joins:
> {code}
> table1.join(table2).join(table2);// "join" could be replaced with
> "leftJoin" and "outerJoin"
> {code}
> where {{table2}} is involved multiple times in this multi-way joins. In this
> case, when a new record from the source topic of {{table2}} is being
> processing, it will send to two children down in the topology and hence may
> resulting in duplicated join results depending on the join types.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-7965) Flaky Test ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
[
https://issues.apache.org/jira/browse/KAFKA-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069288#comment-17069288
]
Matthias J. Sax commented on KAFKA-7965:
[https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1446/testReport/junit/kafka.api/ConsumerBounceTest/testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup/]
> Flaky Test
> ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
>
>
> Key: KAFKA-7965
> URL: https://issues.apache.org/jira/browse/KAFKA-7965
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 1.1.1, 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/21/]
> {quote}java.lang.AssertionError: Received 0, expected at least 68 at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.ConsumerBounceTest.receiveAndCommit(ConsumerBounceTest.scala:557)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1(ConsumerBounceTest.scala:320)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1$adapted(ConsumerBounceTest.scala:319)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup(ConsumerBounceTest.scala:319){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
