[jira] [Assigned] (KAFKA-7983) supporting replication.throttled.replicas in dynamic broker configuration
[ https://issues.apache.org/jira/browse/KAFKA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu reassigned KAFKA-7983: Assignee: (was: lambdaliu) > supporting replication.throttled.replicas in dynamic broker configuration > - > > Key: KAFKA-7983 > URL: https://issues.apache.org/jira/browse/KAFKA-7983 > Project: Kafka > Issue Type: New Feature > Components: core >Reporter: Jun Rao >Priority: Major > > In > [KIP-226|https://cwiki.apache.org/confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration#KIP-226-DynamicBrokerConfiguration-DefaultTopicconfigs], > we added the support to change broker defaults dynamically. However, it > didn't support changing leader.replication.throttled.replicas and > follower.replication.throttled.replicas. These 2 configs were introduced in > [KIP-73|https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas] > and controls the set of topic partitions on which replication throttling > will be engaged. One useful case is to be able to set a default value for > both configs to * to allow throttling to be engaged for all topic partitions. > Currently, the static default value for both configs are ignored for > replication throttling, it would be useful to fix that as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7986) distinguish the logging from different ZooKeeperClient instances
[ https://issues.apache.org/jira/browse/KAFKA-7986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu reassigned KAFKA-7986: Assignee: (was: lambdaliu) > distinguish the logging from different ZooKeeperClient instances > > > Key: KAFKA-7986 > URL: https://issues.apache.org/jira/browse/KAFKA-7986 > Project: Kafka > Issue Type: Improvement >Reporter: Jun Rao >Priority: Major > Labels: newbie > > It's possible for each broker to have more than 1 ZooKeeperClient instance. > For example, SimpleAclAuthorizer creates a separate ZooKeeperClient instance > when configured. It would be useful to distinguish the logging from different > ZooKeeperClient instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7986) distinguish the logging from different ZooKeeperClient instances
[ https://issues.apache.org/jira/browse/KAFKA-7986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu reassigned KAFKA-7986: Assignee: lambdaliu > distinguish the logging from different ZooKeeperClient instances > > > Key: KAFKA-7986 > URL: https://issues.apache.org/jira/browse/KAFKA-7986 > Project: Kafka > Issue Type: Improvement >Reporter: Jun Rao >Assignee: lambdaliu >Priority: Major > Labels: newbie > > It's possible for each broker to have more than 1 ZooKeeperClient instance. > For example, SimpleAclAuthorizer creates a separate ZooKeeperClient instance > when configured. It would be useful to distinguish the logging from different > ZooKeeperClient instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7811) Avoid unnecessary lock acquire when KafkaConsumer commit offsets
[
https://issues.apache.org/jira/browse/KAFKA-7811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lambdaliu updated KAFKA-7811:
-
Description:
In KafkaConsumer#commitSync, we have the following logic:
{code:java}
public void commitAsync(OffsetCommitCallback callback) {
acquireAndEnsureOpen();
try {
commitAsync(subscriptions.allConsumed(), callback);
} finally {
release();
}
}
{code}
This function calls another commitAsync which also call `acquireAndEnsureOpen`.
was:
In KafkaConsumer#commitSync, we have the following logic:
{code:java}
public void commitAsync(OffsetCommitCallback callback) {
acquireAndEnsureOpen();
try {
commitAsync(subscriptions.allConsumed(), callback);
} finally {
release();
}
}
{code}
This function calls another commitAsync which also call `acquireAndEnsureOpen`.
> Avoid unnecessary lock acquire when KafkaConsumer commit offsets
>
>
> Key: KAFKA-7811
> URL: https://issues.apache.org/jira/browse/KAFKA-7811
> Project: Kafka
> Issue Type: Improvement
> Components: clients
>Affects Versions: 0.10.2.2, 0.11.0.3, 1.0.2, 1.1.1, 2.0.1, 2.1.0
>Reporter: lambdaliu
>Assignee: lambdaliu
>Priority: Major
>
> In KafkaConsumer#commitSync, we have the following logic:
> {code:java}
> public void commitAsync(OffsetCommitCallback callback) {
> acquireAndEnsureOpen();
> try {
> commitAsync(subscriptions.allConsumed(), callback);
> } finally {
> release();
> }
> }
> {code}
> This function calls another commitAsync which also call
> `acquireAndEnsureOpen`.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7811) Avoid unnecessary lock acquire when KafkaConsumer commit offsets
lambdaliu created KAFKA-7811:
Summary: Avoid unnecessary lock acquire when KafkaConsumer commit
offsets
Key: KAFKA-7811
URL: https://issues.apache.org/jira/browse/KAFKA-7811
Project: Kafka
Issue Type: Improvement
Components: clients
Affects Versions: 2.1.0, 2.0.1, 1.1.1, 1.0.2, 0.11.0.3, 0.10.2.2
Reporter: lambdaliu
Assignee: lambdaliu
In KafkaConsumer#commitSync, we have the following logic:
{code:java}
public void commitAsync(OffsetCommitCallback callback) {
acquireAndEnsureOpen();
try {
commitAsync(subscriptions.allConsumed(), callback);
} finally {
release();
}
}
{code}
This function calls another commitAsync which also call `acquireAndEnsureOpen`.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7734) Metrics tags should use LinkedHashMap to guarantee ordering
[
https://issues.apache.org/jira/browse/KAFKA-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lambdaliu reassigned KAFKA-7734:
Assignee: lambdaliu
> Metrics tags should use LinkedHashMap to guarantee ordering
> ---
>
> Key: KAFKA-7734
> URL: https://issues.apache.org/jira/browse/KAFKA-7734
> Project: Kafka
> Issue Type: Improvement
> Components: metrics
>Reporter: Guozhang Wang
>Assignee: lambdaliu
>Priority: Major
>
> Today we store metrics tags inside MetricName from various places, and many
> of them are using `HashMap`. However, for metrics reporters like JMXReporter,
> the mBeanName is constructed by looping over `metricName.tags().entrySet()`
> which does not guarantee ordering. This resulted a few places where the
> mBeanName string not as expected, e.g. we document the Streams cache metrics
> as
> {code}
> kafka.streams:type=stream-record-cache-metrics,client-id=([-.\w]+),task-id=([-.\w]+),record-cache-id=([-.\w]+)
> {code}
> However, what I've seen from JMXReporter is, for example:
> {code}
> kafka.streams:type=stream-record-cache-metrics,record-cache-id=all,client-id=streams-saak-test-client-StreamThread-1,task-id=1_3
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7535) KafkaConsumer doesn't report records-lag if isolation.level is read_committed
[
https://issues.apache.org/jira/browse/KAFKA-7535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661789#comment-16661789
]
lambdaliu commented on KAFKA-7535:
--
bug introduced from this [PR|https://github.com/apache/kafka/pull/5192]
> KafkaConsumer doesn't report records-lag if isolation.level is read_committed
> -
>
> Key: KAFKA-7535
> URL: https://issues.apache.org/jira/browse/KAFKA-7535
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 2.0.0
>Reporter: Alexey Vakhrenev
>Assignee: lambdaliu
>Priority: Major
>
> Starting from 2.0.0, {{KafkaConsumer}} doesn't report {{records-lag}} if
> {{isolation.level}} is {{read_committed}}. The last version, where it works
> is {{1.1.1}}.
> The issue can be easily reproduced in {{kafka.api.PlaintextConsumerTest}} by
> adding {{consumerConfig.setProperty("isolation.level", "read_committed")}}
> witin related tests:
> - {{testPerPartitionLagMetricsCleanUpWithAssign}}
> - {{testPerPartitionLagMetricsCleanUpWithSubscribe}}
> - {{testPerPartitionLagWithMaxPollRecords}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7535) KafkaConsumer doesn't report records-lag if isolation.level is read_committed
[
https://issues.apache.org/jira/browse/KAFKA-7535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lambdaliu reassigned KAFKA-7535:
Assignee: lambdaliu
> KafkaConsumer doesn't report records-lag if isolation.level is read_committed
> -
>
> Key: KAFKA-7535
> URL: https://issues.apache.org/jira/browse/KAFKA-7535
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 2.0.0
>Reporter: Alexey Vakhrenev
>Assignee: lambdaliu
>Priority: Major
>
> Starting from 2.0.0, {{KafkaConsumer}} doesn't report {{records-lag}} if
> {{isolation.level}} is {{read_committed}}. The last version, where it works
> is {{1.1.1}}.
> The issue can be easily reproduced in {{kafka.api.PlaintextConsumerTest}} by
> adding {{consumerConfig.setProperty("isolation.level", "read_committed")}}
> witin related tests:
> - {{testPerPartitionLagMetricsCleanUpWithAssign}}
> - {{testPerPartitionLagMetricsCleanUpWithSubscribe}}
> - {{testPerPartitionLagWithMaxPollRecords}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7190) Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589573#comment-16589573 ] lambdaliu commented on KAFKA-7190: -- Hi [~guozhang] As far as I am concerned , the follower also maintains PID cache. The case that Joson described will happen when partition reassignment with leader migrate. > Under low traffic conditions purging repartition topics cause WARN statements > about UNKNOWN_PRODUCER_ID > - > > Key: KAFKA-7190 > URL: https://issues.apache.org/jira/browse/KAFKA-7190 > Project: Kafka > Issue Type: Improvement > Components: core, streams >Affects Versions: 1.1.0, 1.1.1 >Reporter: Bill Bejeck >Assignee: lambdaliu >Priority: Major > > When a streams application has little traffic, then it is possible that > consumer purging would delete > even the last message sent by a producer (i.e., all the messages sent by > this producer have been consumed and committed), and as a result, the broker > would delete that producer's ID. The next time when this producer tries to > send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, > this error is retriable: the producer would just get a new producer id and > retries, and then this time it will succeed. > > Possible fixes could be on the broker side, i.e., delaying the deletion of > the produderIDs for a more extended period or on the streams side developing > a more conservative approach to deleting offsets from repartition topics > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7190) Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16586903#comment-16586903 ] lambdaliu commented on KAFKA-7190: -- Hi [~mjsax], Changing the log level to DEBUG can't avoid the UNKNOWN_PRODUCER error which can cause one more Request-Response to reset producer epoch and sequence. > Under low traffic conditions purging repartition topics cause WARN statements > about UNKNOWN_PRODUCER_ID > - > > Key: KAFKA-7190 > URL: https://issues.apache.org/jira/browse/KAFKA-7190 > Project: Kafka > Issue Type: Improvement > Components: core, streams >Affects Versions: 1.1.0, 1.1.1 >Reporter: Bill Bejeck >Assignee: lambdaliu >Priority: Major > > When a streams application has little traffic, then it is possible that > consumer purging would delete > even the last message sent by a producer (i.e., all the messages sent by > this producer have been consumed and committed), and as a result, the broker > would delete that producer's ID. The next time when this producer tries to > send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, > this error is retriable: the producer would just get a new producer id and > retries, and then this time it will succeed. > > Possible fixes could be on the broker side, i.e., delaying the deletion of > the produderIDs for a more extended period or on the streams side developing > a more conservative approach to deleting offsets from repartition topics > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7190) Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16586886#comment-16586886 ] lambdaliu commented on KAFKA-7190: -- Hi [~hachikuji], sorry for the later reply. I have thought about the solution you suggested last time, and found it's not easy to reset sequence to 0 for each transaction. Because the broker may receive a new transactional produce request before the last complete transaction's EndTxnMaker request. So we would better cache all PID in memory until it expired. For the idea of soft delete records which beyond the LSO, it's easy to implement. But the deletion of retention time and retention size breach may still delete the segment which contain active transaction. For this case, maybe we can use snapshot file to save the active PID. With this change we can always recover PID which have active transaction from log. For the KIP you posted seems great and I am glad to work on it. Thanks. > Under low traffic conditions purging repartition topics cause WARN statements > about UNKNOWN_PRODUCER_ID > - > > Key: KAFKA-7190 > URL: https://issues.apache.org/jira/browse/KAFKA-7190 > Project: Kafka > Issue Type: Improvement > Components: core, streams >Affects Versions: 1.1.0, 1.1.1 >Reporter: Bill Bejeck >Assignee: lambdaliu >Priority: Major > > When a streams application has little traffic, then it is possible that > consumer purging would delete > even the last message sent by a producer (i.e., all the messages sent by > this producer have been consumed and committed), and as a result, the broker > would delete that producer's ID. The next time when this producer tries to > send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, > this error is retriable: the producer would just get a new producer id and > retries, and then this time it will succeed. > > Possible fixes could be on the broker side, i.e., delaying the deletion of > the produderIDs for a more extended period or on the streams side developing > a more conservative approach to deleting offsets from repartition topics > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5587) Processor got uncaught exception: NullPointerException
[ https://issues.apache.org/jira/browse/KAFKA-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567718#comment-16567718 ] lambdaliu commented on KAFKA-5587: -- Hi [~rsivaram], I found this issue still appears in version 0.10.2.2, may be we should port the fix to 0.10.2 branch. > Processor got uncaught exception: NullPointerException > -- > > Key: KAFKA-5587 > URL: https://issues.apache.org/jira/browse/KAFKA-5587 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.10.1.1, 0.10.2.1 >Reporter: Dan >Assignee: Rajini Sivaram >Priority: Major > Fix For: 0.11.0.1, 1.0.0 > > > [2017-07-12 21:56:39,964] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.NullPointerException > at > kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:490) > at > kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:487) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > kafka.network.Processor.processCompletedReceives(SocketServer.scala:487) > at kafka.network.Processor.run(SocketServer.scala:417) > at java.lang.Thread.run(Thread.java:745) > Anyone knows the cause of this exception? What's the effect of it? > When this exception occurred, the log also showed that the broker was > frequently shrinking ISR to itself. Are these two things interrelated? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-5587) Processor got uncaught exception: NullPointerException
[ https://issues.apache.org/jira/browse/KAFKA-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu updated KAFKA-5587: - Affects Version/s: 0.10.2.1 > Processor got uncaught exception: NullPointerException > -- > > Key: KAFKA-5587 > URL: https://issues.apache.org/jira/browse/KAFKA-5587 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.10.1.1, 0.10.2.1 >Reporter: Dan >Assignee: Rajini Sivaram >Priority: Major > Fix For: 0.11.0.1, 1.0.0 > > > [2017-07-12 21:56:39,964] ERROR Processor got uncaught exception. > (kafka.network.Processor) > java.lang.NullPointerException > at > kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:490) > at > kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:487) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > kafka.network.Processor.processCompletedReceives(SocketServer.scala:487) > at kafka.network.Processor.run(SocketServer.scala:417) > at java.lang.Thread.run(Thread.java:745) > Anyone knows the cause of this exception? What's the effect of it? > When this exception occurred, the log also showed that the broker was > frequently shrinking ISR to itself. Are these two things interrelated? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7190) Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564694#comment-16564694 ] lambdaliu commented on KAFKA-7190: -- Hi [~hachikuji], What do you think about the solution of the problem? Looking forward to your opinion. Thanks! > Under low traffic conditions purging repartition topics cause WARN statements > about UNKNOWN_PRODUCER_ID > - > > Key: KAFKA-7190 > URL: https://issues.apache.org/jira/browse/KAFKA-7190 > Project: Kafka > Issue Type: Improvement > Components: core, streams >Affects Versions: 1.1.0, 1.1.1 >Reporter: Bill Bejeck >Assignee: lambdaliu >Priority: Major > > When a streams application has little traffic, then it is possible that > consumer purging would delete > even the last message sent by a producer (i.e., all the messages sent by > this producer have been consumed and committed), and as a result, the broker > would delete that producer's ID. The next time when this producer tries to > send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, > this error is retriable: the producer would just get a new producer id and > retries, and then this time it will succeed. > > Possible fixes could be on the broker side, i.e., delaying the deletion of > the produderIDs for a more extended period or on the streams side developing > a more conservative approach to deleting offsets from repartition topics > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7190) Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558265#comment-16558265 ] lambdaliu commented on KAFKA-7190: -- Hello [~guozhang]. My team developed a cloud version Kafka and I am familiar with the broker. So I think probably I can solve this issue. When we remove the head of the log, we take the bellowing steps in ProducerStateManager.truncateHead : 1. clean producerId whose last offset smaller than log start offset 2. remove procducerId's BatchMetadata which have a last offset smaller than log start offset 3. remove ongoing transaction whose producerId remove in step 1. 4. remove unreplicated transaction whose last offset smaller than log start offset 5. update lastMapOffset to log start offset if lasterMapoffset is smaller than log start offset 6. delete snapshot file older than the new log start offset As you suggested, we can delay the deletion of producer ID until it expired. We can also delay the step 2 and step 3 to that time. For the old snapshot file in step 6, we can rely on the period called function deleteSnapshotsAfterRecoveryPointCheckpoint to delete it. And when loading producer state from snapshot file we may not drop the producerId whose last offset smaller than log start offset. So we just need do step 4 and step 5 when remove the head of the log. For the additional PID expiration config, is there any reason to add it? if it is reasonable, I will add it. > Under low traffic conditions purging repartition topics cause WARN statements > about UNKNOWN_PRODUCER_ID > - > > Key: KAFKA-7190 > URL: https://issues.apache.org/jira/browse/KAFKA-7190 > Project: Kafka > Issue Type: Improvement > Components: core, streams >Affects Versions: 1.1.0, 1.1.1 >Reporter: Bill Bejeck >Assignee: lambdaliu >Priority: Major > > When a streams application has little traffic, then it is possible that > consumer purging would delete > even the last message sent by a producer (i.e., all the messages sent by > this producer have been consumed and committed), and as a result, the broker > would delete that producer's ID. The next time when this producer tries to > send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, > this error is retriable: the producer would just get a new producer id and > retries, and then this time it will succeed. > > Possible fixes could be on the broker side, i.e., delaying the deletion of > the produderIDs for a more extended period or on the streams side developing > a more conservative approach to deleting offsets from repartition topics > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7190) Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555730#comment-16555730 ] lambdaliu commented on KAFKA-7190: -- Hello [~mjsax] [~guozhang], sorry for the late reply. I also agree to fix this issue on the broker side. As [~guozhang] said, we can delaying the deletion of producer ID to resolve this issue. The problem here is how long to delay the deletion. Is 60 seconds OK? > Under low traffic conditions purging repartition topics cause WARN statements > about UNKNOWN_PRODUCER_ID > - > > Key: KAFKA-7190 > URL: https://issues.apache.org/jira/browse/KAFKA-7190 > Project: Kafka > Issue Type: Improvement > Components: core, streams >Affects Versions: 1.1.0, 1.1.1 >Reporter: Bill Bejeck >Assignee: lambdaliu >Priority: Major > > When a streams application has little traffic, then it is possible that > consumer purging would delete > even the last message sent by a producer (i.e., all the messages sent by > this producer have been consumed and committed), and as a result, the broker > would delete that producer's ID. The next time when this producer tries to > send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, > this error is retriable: the producer would just get a new producer id and > retries, and then this time it will succeed. > > Possible fixes could be on the broker side, i.e., delaying the deletion of > the produderIDs for a more extended period or on the streams side developing > a more conservative approach to deleting offsets from repartition topics > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7190) Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu reassigned KAFKA-7190: Assignee: lambdaliu > Under low traffic conditions purging repartition topics cause WARN statements > about UNKNOWN_PRODUCER_ID > - > > Key: KAFKA-7190 > URL: https://issues.apache.org/jira/browse/KAFKA-7190 > Project: Kafka > Issue Type: Improvement > Components: core, streams >Affects Versions: 1.1.0, 1.1.1 >Reporter: Bill Bejeck >Assignee: lambdaliu >Priority: Major > > When a streams application has little traffic, then it is possible that > consumer purging would delete > even the last message sent by a producer (i.e., all the messages sent by > this producer have been consumed and committed), and as a result, the broker > would delete that producer's ID. The next time when this producer tries to > send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, > this error is retriable: the producer would just get a new producer id and > retries, and then this time it will succeed. > > Possible fixes could be on the broker side, i.e., delaying the deletion of > the produderIDs for a more extended period or on the streams side developing > a more conservative approach to deleting offsets from repartition topics > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-7160) Add check for group ID length
[ https://issues.apache.org/jira/browse/KAFKA-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549162#comment-16549162 ] lambdaliu edited comment on KAFKA-7160 at 7/19/18 12:02 PM: Hi, [~Bitswazsky] [~a.bircakovic] Sorry for the late reply. I have tested consumer with 4kB length group ID to consume message without error. In my humble opinion, 256B is enough for most use case. was (Author: lambdaliu): Hi, [~Bitswazsky] [~a.bircakovic] Sorry for the late reply. I have tested consumer with 4kB length group ID to consume message without error. I think 256B is enough for most use case. > Add check for group ID length > - > > Key: KAFKA-7160 > URL: https://issues.apache.org/jira/browse/KAFKA-7160 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: lambdaliu >Assignee: Sayan Biswas >Priority: Minor > Labels: newbie > > We should limit the length of the group ID, because other system(such as > monitor system) would use the group ID when we using kafka in production > environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7160) Add check for group ID length
[ https://issues.apache.org/jira/browse/KAFKA-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549162#comment-16549162 ] lambdaliu commented on KAFKA-7160: -- Hi, [~Bitswazsky] [~a.bircakovic] Sorry for the late reply. I have tested consumer with 4kB length group ID to consume message without error. I think 256B is enough for most use case. > Add check for group ID length > - > > Key: KAFKA-7160 > URL: https://issues.apache.org/jira/browse/KAFKA-7160 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: lambdaliu >Assignee: Sayan Biswas >Priority: Minor > Labels: newbie > > We should limit the length of the group ID, because other system(such as > monitor system) would use the group ID when we using kafka in production > environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7160) Add check for group ID length
lambdaliu created KAFKA-7160: Summary: Add check for group ID length Key: KAFKA-7160 URL: https://issues.apache.org/jira/browse/KAFKA-7160 Project: Kafka Issue Type: Improvement Components: core Reporter: lambdaliu We should limit the length of the group ID, because other system(such as monitor system) would use the group ID when we using kafka in production environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5409) Providing a custom client-id to the ConsoleProducer tool
[
https://issues.apache.org/jira/browse/KAFKA-5409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534715#comment-16534715
]
lambdaliu commented on KAFKA-5409:
--
Hi [~ppatierno],
We also encounter this issue when using 'kafka-console-producer.sh' to test
some feature.
And we found this issue can be fixed if we replace the line
{code:java}
props.put("client.id", "console-producer"){code}
in ConsoleProducer.scala to
{code:java}
props.putIfAbsent("client.id", "console-producer"){code}
> Providing a custom client-id to the ConsoleProducer tool
>
>
> Key: KAFKA-5409
> URL: https://issues.apache.org/jira/browse/KAFKA-5409
> Project: Kafka
> Issue Type: Improvement
> Components: tools
>Affects Versions: 1.1.0, 1.0.1
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>Priority: Minor
>
> Hi,
> I see that the client-id properties for the ConsoleProducer tool is always
> "console-producer". It could be useful having it as parameter on the command
> line or generating a random one like happens for the ConsolerConsumer.
> If it makes sense to you, I can work on that.
> Thanks,
> Paolo.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-5409) Providing a custom client-id to the ConsoleProducer tool
[ https://issues.apache.org/jira/browse/KAFKA-5409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu updated KAFKA-5409: - Affects Version/s: 1.1.0 1.0.1 > Providing a custom client-id to the ConsoleProducer tool > > > Key: KAFKA-5409 > URL: https://issues.apache.org/jira/browse/KAFKA-5409 > Project: Kafka > Issue Type: Improvement > Components: tools >Affects Versions: 1.1.0, 1.0.1 >Reporter: Paolo Patierno >Assignee: Paolo Patierno >Priority: Minor > > Hi, > I see that the client-id properties for the ConsoleProducer tool is always > "console-producer". It could be useful having it as parameter on the command > line or generating a random one like happens for the ConsolerConsumer. > If it makes sense to you, I can work on that. > Thanks, > Paolo. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7120) When Connect throws CONFLICT error for REST requests, it will help to see more details
[ https://issues.apache.org/jira/browse/KAFKA-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528659#comment-16528659 ] lambdaliu commented on KAFKA-7120: -- Hi, I am a newbie. I want to get familiar with the contribution process by solve this issue. We can add the path, method and body of the request to the message field of ConnectRestException to indicate which request can't be completed. [~gwenshap] Do you think this is a suitable solution for this issue? > When Connect throws CONFLICT error for REST requests, it will help to see > more details > -- > > Key: KAFKA-7120 > URL: https://issues.apache.org/jira/browse/KAFKA-7120 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Gwen Shapira >Assignee: lambdaliu >Priority: Critical > > Right now, we throw: > throw new ConnectRestException(Response.Status.CONFLICT.getStatusCode(), > "Cannot complete request because of a conflicting operation (e.g. worker > rebalance)"); > There's no information about WHICH request can't be completed. It will help > to know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7120) When Connect throws CONFLICT error for REST requests, it will help to see more details
[ https://issues.apache.org/jira/browse/KAFKA-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu updated KAFKA-7120: - Component/s: KafkaConnect > When Connect throws CONFLICT error for REST requests, it will help to see > more details > -- > > Key: KAFKA-7120 > URL: https://issues.apache.org/jira/browse/KAFKA-7120 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Gwen Shapira >Assignee: lambdaliu >Priority: Critical > > Right now, we throw: > throw new ConnectRestException(Response.Status.CONFLICT.getStatusCode(), > "Cannot complete request because of a conflicting operation (e.g. worker > rebalance)"); > There's no information about WHICH request can't be completed. It will help > to know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7120) When Connect throws CONFLICT error for REST requests, it will help to see more details
[ https://issues.apache.org/jira/browse/KAFKA-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu updated KAFKA-7120: - Labels: kafkaconnect (was: connect) > When Connect throws CONFLICT error for REST requests, it will help to see > more details > -- > > Key: KAFKA-7120 > URL: https://issues.apache.org/jira/browse/KAFKA-7120 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Gwen Shapira >Assignee: lambdaliu >Priority: Critical > > Right now, we throw: > throw new ConnectRestException(Response.Status.CONFLICT.getStatusCode(), > "Cannot complete request because of a conflicting operation (e.g. worker > rebalance)"); > There's no information about WHICH request can't be completed. It will help > to know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7120) When Connect throws CONFLICT error for REST requests, it will help to see more details
[ https://issues.apache.org/jira/browse/KAFKA-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu updated KAFKA-7120: - Labels: (was: kafkaconnect) > When Connect throws CONFLICT error for REST requests, it will help to see > more details > -- > > Key: KAFKA-7120 > URL: https://issues.apache.org/jira/browse/KAFKA-7120 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Gwen Shapira >Assignee: lambdaliu >Priority: Critical > > Right now, we throw: > throw new ConnectRestException(Response.Status.CONFLICT.getStatusCode(), > "Cannot complete request because of a conflicting operation (e.g. worker > rebalance)"); > There's no information about WHICH request can't be completed. It will help > to know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7120) When Connect throws CONFLICT error for REST requests, it will help to see more details
[ https://issues.apache.org/jira/browse/KAFKA-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu reassigned KAFKA-7120: Assignee: lambdaliu > When Connect throws CONFLICT error for REST requests, it will help to see > more details > -- > > Key: KAFKA-7120 > URL: https://issues.apache.org/jira/browse/KAFKA-7120 > Project: Kafka > Issue Type: Improvement >Reporter: Gwen Shapira >Assignee: lambdaliu >Priority: Critical > Labels: connect > > Right now, we throw: > throw new ConnectRestException(Response.Status.CONFLICT.getStatusCode(), > "Cannot complete request because of a conflicting operation (e.g. worker > rebalance)"); > There's no information about WHICH request can't be completed. It will help > to know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7120) When Connect throws CONFLICT error for REST requests, it will help to see more details
[ https://issues.apache.org/jira/browse/KAFKA-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lambdaliu updated KAFKA-7120: - Labels: connect (was: ) > When Connect throws CONFLICT error for REST requests, it will help to see > more details > -- > > Key: KAFKA-7120 > URL: https://issues.apache.org/jira/browse/KAFKA-7120 > Project: Kafka > Issue Type: Improvement >Reporter: Gwen Shapira >Priority: Critical > Labels: connect > > Right now, we throw: > throw new ConnectRestException(Response.Status.CONFLICT.getStatusCode(), > "Cannot complete request because of a conflicting operation (e.g. worker > rebalance)"); > There's no information about WHICH request can't be completed. It will help > to know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
