[jira] [Created] (KAFKA-9149) Avoid temp byte array creation when use ByteBufferSerializer
chenxu created KAFKA-9149:
-
Summary: Avoid temp byte array creation when use
ByteBufferSerializer
Key: KAFKA-9149
URL: https://issues.apache.org/jira/browse/KAFKA-9149
Project: Kafka
Issue Type: Improvement
Components: clients
Reporter: chenxu
Code in ByteBufferSerializer#serialize like this
{code:java}
public byte[] serialize(String topic, ByteBuffer data) {
if (data == null)
return null;
data.rewind();
if (data.hasArray()) {
byte[] arr = data.array();
if (data.arrayOffset() == 0 && arr.length == data.remaining()) {
return arr;
}
}
byte[] ret = new byte[data.remaining()];
data.get(ret, 0, ret.length);
data.rewind();
return ret;
}
{code}
Temp byte array will be created when use with DirectByteBuffer, how about
declare a method such as serialize2Buffer and return the ByteBuffer directly ?
This can improve GC a lot in KafkaProducer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9148) Consider forking RocksDB for Streams
[
https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968119#comment-16968119
]
Sophie Blee-Goldman edited comment on KAFKA-9148 at 11/6/19 6:37 AM:
-
{quote}seems like this is the kind of problem that the RocksDB team _would_
care about
{quote}
I agree, which is why I found it odd that it's been two months since the
regression issue was first brought to their notice, and the most recent comment
on the issue is actually just the issue's author still trying to get attention
on it from the rocksdb folks with three weeks of silence since :)
(To be fair that doesn't mean they definitely aren't investigating it
currently, but I'll just say that response level is representative of other
major issues in the past – and any Java-specific performance problems seem to
almost always get dropped on the floor)
I should clarify that I'm not necessarily advocating strongly for forking it
_right now_ – but if we find ourselves coming back to this list and adding more
things, or in a place where the lack of some feature or Java API is holding
Streams back, then we should consider it in a serious way.
was (Author: ableegoldman):
> seems like this is the kind of problem that the RocksDB team _would_ care
>about
I agree, which is why I found it odd that it's been two months since the
regression issue was first brought to their notice, and the most recent comment
on the issue is actually just the issue's author still trying to get attention
on it from the rocksdb folks with three weeks of silence since :)
(To be fair that doesn't mean they definitely aren't investigating it
currently, but I'll just say that response level is representative of other
major issues in the past – and any Java-specific performance problems seem to
almost always get dropped on the floor)
I should clarify that I'm not necessarily advocating strongly for forking it
_right now_ – but if we find ourselves coming back to this list and adding more
things, or in a place where the lack of some feature or Java API is holding
Streams back, then we should consider it in a serious way.
> Consider forking RocksDB for Streams
> -
>
> Key: KAFKA-9148
> URL: https://issues.apache.org/jira/browse/KAFKA-9148
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Sophie Blee-Goldman
>Priority: Major
>
> We recently upgraded our RocksDB dependency to 5.18 for its memory-management
> abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately,
> someone from Flink recently discovered a ~8% [performance
> regression|https://github.com/facebook/rocksdb/issues/5774] that exists in
> all versions 5.18+ (up through the current newest version, 6.2.2). Flink was
> able to react to this by downgrading to 5.17 and [picking the
> WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their
> fork (fRocksDB).
> Due to this and other reasons enumerated below, we should consider also
> forking our own RocksDB for Streams.
> Pros:
> * We can avoid passing sudden breaking changes on to our users, such removal
> of methods with no deprecation period (see discussion on KAFKA-8897)
> * We can pick whichever version has the best performance for our needs, and
> pick over any new features, metrics, etc that we need to use rather than
> being forced to upgrade (and breaking user code, introducing regression, etc)
> * The Java API seems to be a very low priority to the rocksdb folks.
> ** They leave out critical functionality, features, and configuration
> options that have been in the c++ API for a very long time
> ** Those that do make it over often have random gaps in the API such as
> setters but no getters (see [rocksdb PR
> #5186|https://github.com/facebook/rocksdb/pull/5186])
> ** Others are poorly designed and require too many trips across the JNI,
> making otherwise incredibly useful features prohibitively expensive.
> *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could
> significantly improve the performance of session windows. This is trivial to
> do but given the high performance cost of crossing the jni, it is currently
> only practical to use a c++ comparator
> *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not
> currently used by Streams but a commonly requested feature, and may also
> allow improved range queries
> ** Even when an external contributor develops a solution for poorly
> performing Java functionality and helpfully tries to contribute their patch
> back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR
> #2283|https://github.com/facebook/rocksdb/pull/2283])
> Cons:
> * more work
> Given that we rarely upgrade the Rocks dependency, use
[jira] [Commented] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968119#comment-16968119 ] Sophie Blee-Goldman commented on KAFKA-9148: > seems like this is the kind of problem that the RocksDB team _would_ care >about I agree, which is why I found it odd that it's been two months since the regression issue was first brought to their notice, and the most recent comment on the issue is actually just the issue's author still trying to get attention on it from the rocksdb folks with three weeks of silence since :) (To be fair that doesn't mean they definitely aren't investigating it currently, but I'll just say that response level is representative of other major issues in the past – and any Java-specific performance problems seem to almost always get dropped on the floor) I should clarify that I'm not necessarily advocating strongly for forking it _right now_ – but if we find ourselves coming back to this list and adding more things, or in a place where the lack of some feature or Java API is holding Streams back, then we should consider it in a serious way. > Consider forking RocksDB for Streams > - > > Key: KAFKA-9148 > URL: https://issues.apache.org/jira/browse/KAFKA-9148 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Sophie Blee-Goldman >Priority: Major > > We recently upgraded our RocksDB dependency to 5.18 for its memory-management > abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, > someone from Flink recently discovered a ~8% [performance > regression|https://github.com/facebook/rocksdb/issues/5774] that exists in > all versions 5.18+ (up through the current newest version, 6.2.2). Flink was > able to react to this by downgrading to 5.17 and [picking the > WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their > fork (fRocksDB). > Due to this and other reasons enumerated below, we should consider also > forking our own RocksDB for Streams. > Pros: > * We can avoid passing sudden breaking changes on to our users, such removal > of methods with no deprecation period (see discussion on KAFKA-8897) > * We can pick whichever version has the best performance for our needs, and > pick over any new features, metrics, etc that we need to use rather than > being forced to upgrade (and breaking user code, introducing regression, etc) > * The Java API seems to be a very low priority to the rocksdb folks. > ** They leave out critical functionality, features, and configuration > options that have been in the c++ API for a very long time > ** Those that do make it over often have random gaps in the API such as > setters but no getters (see [rocksdb PR > #5186|https://github.com/facebook/rocksdb/pull/5186]) > ** Others are poorly designed and require too many trips across the JNI, > making otherwise incredibly useful features prohibitively expensive. > *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could > significantly improve the performance of session windows. This is trivial to > do but given the high performance cost of crossing the jni, it is currently > only practical to use a c++ comparator > *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not > currently used by Streams but a commonly requested feature, and may also > allow improved range queries > ** Even when an external contributor develops a solution for poorly > performing Java functionality and helpfully tries to contribute their patch > back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR > #2283|https://github.com/facebook/rocksdb/pull/2283]) > Cons: > * more work > Given that we rarely upgrade the Rocks dependency, use only some fraction of > its features, and would need or want to make only minimal changes ourselves, > it seems like we could actually get away with very little extra work by > forking rocksdb. Note that as of this writing the frocksdb repo has only > needed to open 5 PRs on top of the actual rocksdb (two of them trivial). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9148: --- Description: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone from Flink recently discovered a ~8% [performance regression|https://github.com/facebook/rocksdb/issues/5774] that exists in all versions 5.18+ (up through the current newest version, 6.2.2). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. Note that as of this writing the frocksdb repo has only needed to open 5 PRs on top of the actual rocksdb (two of them trivial). was: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone from Flink recently discovered a ~8% performance regression that exists in all versions 5.18+ (up through the current newest version, 6.2.2). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored
[jira] [Commented] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968097#comment-16968097 ] John Roesler commented on KAFKA-9148: - Thanks for bringing this up, Sophie. It does seem like this might be the best solution given the points you raise, but I'm still wary of the burden of maintaining it. This might be a silly thought, but would it make sense (and comply with our license) to just use fRocksDB? Also, there are several good reasons you provided, but specifically on the performance regression, it seems like this is the kind of problem that the RocksDB team _would_ care about, and would probably fix soon(ish). Is there a reason to think that they wouldn't? Not objections, just questions. Thanks again! > Consider forking RocksDB for Streams > - > > Key: KAFKA-9148 > URL: https://issues.apache.org/jira/browse/KAFKA-9148 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Sophie Blee-Goldman >Priority: Major > > We recently upgraded our RocksDB dependency to 5.18 for its memory-management > abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, > someone from Flink recently discovered a ~8% performance regression that > exists in all versions 5.18+ (up through the current newest version, 6.2.2). > Flink was able to react to this by downgrading to 5.17 and [picking the > WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their > fork (fRocksDB). > Due to this and other reasons enumerated below, we should consider also > forking our own RocksDB for Streams. > Pros: > * We can avoid passing sudden breaking changes on to our users, such removal > of methods with no deprecation period (see discussion on KAFKA-8897) > * We can pick whichever version has the best performance for our needs, and > pick over any new features, metrics, etc that we need to use rather than > being forced to upgrade (and breaking user code, introducing regression, etc) > * The Java API seems to be a very low priority to the rocksdb folks. > ** They leave out critical functionality, features, and configuration > options that have been in the c++ API for a very long time > ** Those that do make it over often have random gaps in the API such as > setters but no getters (see [rocksdb PR > #5186|https://github.com/facebook/rocksdb/pull/5186]) > ** Others are poorly designed and require too many trips across the JNI, > making otherwise incredibly useful features prohibitively expensive. > *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could > significantly improve the performance of session windows. This is trivial to > do but given the high performance cost of crossing the jni, it is currently > only practical to use a c++ comparator > *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not > currently used by Streams but a commonly requested feature, and may also > allow improved range queries > ** Even when an external contributor develops a solution for poorly > performing Java functionality and helpfully tries to contribute their patch > back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR > #2283|https://github.com/facebook/rocksdb/pull/2283]) > Cons: > * more work > Given that we rarely upgrade the Rocks dependency, use only some fraction of > its features, and would need or want to make only minimal changes ourselves, > it seems like we could actually get away with very little extra work by > forking rocksdb. Note that as of this writing the frocksdb repo has only > needed to open 5 PRs on top of the actual rocksdb (two of them trivial). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9148: --- Description: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone from Flink recently discovered a ~8% performance regression that exists in all versions 5.18+ (up through the current newest version, 6.2.2). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. Note that as of this writing the frocksdb repo has only needed to open 5 PRs on top of the actual rocksdb (two of them trivial). was: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone from Flink recently discovered a ~8% performance regression that exists in all versions 5.18+ (up through the current newest version, 6.2.2). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR
[jira] [Updated] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9148: --- Description: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone from Flink recently discovered a ~8% performance regression that exists in all versions 5.18+ (up through the current newest version, 6.2.2). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. was: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone from Flink recently discovered a ~8% performance regression that exists in all versions 5.18+ (up through the current newest version, 6.2.2). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some
[jira] [Updated] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9148: --- Description: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely WriteBufferManager, see KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. was: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (WriteBufferManager -- KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only
[jira] [Updated] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9148: --- Description: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (up through the current newest version, 6.2.2). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. was: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would
[jira] [Updated] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9148: --- Description: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. was: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely WriteBufferManager, see KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to
[jira] [Updated] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9148: --- Description: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone from Flink recently discovered a ~8% performance regression that exists in all versions 5.18+ (up through the current newest version, 6.2.2). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. was: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (namely the WriteBufferManager, see KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (up through the current newest version, 6.2.2). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction
[jira] [Updated] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9148: --- Description: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (WriteBufferManager -- KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and [picking the WriteBufferManage|https://github.com/dataArtisans/frocksdb/pull/4]r to their fork (fRocksDB). Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. was: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (WriteBufferManager -- KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and picking the WriteBufferManager to their fork, FRocksDB. Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get
[jira] [Commented] (KAFKA-9137) Maintenance of FetchSession cache causing FETCH_SESSION_ID_NOT_FOUND in live sessions
[
https://issues.apache.org/jira/browse/KAFKA-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968003#comment-16968003
]
ASF GitHub Bot commented on KAFKA-9137:
---
cmccabe commented on pull request #7640: KAFKA-9137: FETCH_SESSION_ID_NOT_FOUND
caused by incorrect FetchSessionCache eviction
URL: https://github.com/apache/kafka/pull/7640
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Maintenance of FetchSession cache causing FETCH_SESSION_ID_NOT_FOUND in live
> sessions
> -
>
> Key: KAFKA-9137
> URL: https://issues.apache.org/jira/browse/KAFKA-9137
> Project: Kafka
> Issue Type: Bug
> Components: core
>Reporter: Lucas Bradstreet
>Priority: Major
>
> We have recently seen cases where brokers end up in a bad state where fetch
> session evictions occur at a high rate (> 16 per second) after a roll. This
> increase in eviction rate included the following pattern in our logs:
>
> {noformat}
> broker 6: October 31st 2019, 17:52:45.496 Created a new incremental
> FetchContext for session id 2046264334, epoch 9790: added (), updated (),
> removed ()
> broker 6: October 31st 2019, 17:52:45.496 Created a new incremental
> FetchContext for session id 2046264334, epoch 9791: added (), updated (),
> removed () broker 6: October 31st 2019, 17:52:45.500 Created a new
> incremental FetchContext for session id 2046264334, epoch 9792: added (),
> updated (lkc-7nv6o_tenant_soak_topic_144p-67), removed ()
> broker 6: October 31st 2019, 17:52:45.501 Created a new incremental
> FetchContext for session id 2046264334, epoch 9793: added (), updated
> (lkc-7nv6o_tenant_soak_topic_144p-59, lkc-7nv6o_tenant_soak_topic_144p-123,
> lkc-7nv6o_tenant_soak_topic_144p-11, lkc-7nv6o_tenant_soak_topic_144p-3,
> lkc-7nv6o_tenant_soak_topic_144p-67, lkc-7nv6o_tenant_soak_topic_144p-115),
> removed ()
> broker 6: October 31st 2019, 17:52:45.501 Evicting stale FetchSession
> 2046264334.
> broker 6: October 31st 2019, 17:52:45.502 Session error for 2046264334: no
> such session ID found.
> broker 4: October 31st 2019, 17:52:45.813 [ReplicaFetcher replicaId=4,
> leaderId=6, fetcherId=0] Node 6 was unable to process the fetch request with
> (sessionId=2046264334, epoch=9793): FETCH_SESSION_ID_NOT_FOUND.
> {noformat}
> This pattern appears to be problematic for two reasons. Firstly, the replica
> fetcher for broker 4 was clearly able to send multiple incremental fetch
> requests to broker 6, and receive replies, and did so right up to the point
> where broker 6 evicted its fetch session within milliseconds of multiple
> fetch requests. The second problem is that replica fetchers are considered
> privileged for the fetch session cache, and should not be evicted by consumer
> fetch sessions. This cluster only has 12 brokers and 1000 fetch session cache
> slots (the default for max.incremental.fetch.session.cache.slots), and it
> thus very unlikely that this session should have been evicted by another
> replica fetcher session.
> This cluster also appears to be causing cycles of fetch session evictions
> where the cluster never stabilizes into a state where fetch sessions are not
> evicted. The above logs are the best example I could find of a case where a
> session clearly should not have been evicted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9148: --- Description: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (WriteBufferManager -- KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and picking the WriteBufferManager to their fork, FRocksDB. Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows. This is trivial to do but given the high performance cost of crossing the jni, it is currently only practical to use a c++ comparator *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. was: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (WriteBufferManager -- KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and picking the WriteBufferManager to their fork, FRocksDB. Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. > Consider forking RocksDB for Streams > - > > Key: KAFKA-9148 >
[jira] [Updated] (KAFKA-9148) Consider forking RocksDB for Streams
[ https://issues.apache.org/jira/browse/KAFKA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-9148: --- Description: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (WriteBufferManager -- KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and picking the WriteBufferManager to their fork, FRocksDB. Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows *** [Prefix Seek|https://github.com/facebook/rocksdb/issues/6004]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. was: We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (WriteBufferManager -- KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and picking the WriteBufferManager to their fork, FRocksDB. Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|[https://github.com/facebook/rocksdb/issues/538#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows *** [Prefix Seek|[https://github.com/facebook/rocksdb/issues/6004]]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. > Consider forking RocksDB for Streams > - > > Key: KAFKA-9148 > URL: https://issues.apache.org/jira/browse/KAFKA-9148 > Project:
[jira] [Created] (KAFKA-9148) Consider forking RocksDB for Streams
Sophie Blee-Goldman created KAFKA-9148: -- Summary: Consider forking RocksDB for Streams Key: KAFKA-9148 URL: https://issues.apache.org/jira/browse/KAFKA-9148 Project: Kafka Issue Type: Improvement Components: streams Reporter: Sophie Blee-Goldman We recently upgraded our RocksDB dependency to 5.18 for its memory-management abilities (WriteBufferManager -- KAFKA-8215). Unfortunately, someone recently discovered a ~8% performance regression that exists in all versions 5.18+ (latest being 6.2.2 as of now). Flink was able to react to this by downgrading to 5.17 and picking the WriteBufferManager to their fork, FRocksDB. Due to this and other reasons enumerated below, we should consider also forking our own RocksDB for Streams. Pros: * We can avoid passing sudden breaking changes on to our users, such removal of methods with no deprecation period (see discussion on KAFKA-8897) * We can pick whichever version has the best performance for our needs, and pick over any new features, metrics, etc that we need to use rather than being forced to upgrade (and breaking user code, introducing regression, etc) * The Java API seems to be a very low priority to the rocksdb folks. ** They leave out critical functionality, features, and configuration options that have been in the c++ API for a very long time ** Those that do make it over often have random gaps in the API such as setters but no getters (see [rocksdb PR #5186|https://github.com/facebook/rocksdb/pull/5186]) ** Others are poorly designed and require too many trips across the JNI, making otherwise incredibly useful features prohibitively expensive. *** [Custom comparator|[https://github.com/facebook/rocksdb/issues/538#issuecomment-83145980]]: a custom comparator could significantly improve the performance of session windows *** [Prefix Seek|[https://github.com/facebook/rocksdb/issues/6004]]: not currently used by Streams but a commonly requested feature, and may also allow improved range queries ** Even when an external contributor develops a solution for poorly performing Java functionality and helpfully tries to contribute their patch back to rocksdb, it gets ignored by the rocksdb people ([rocksdb PR #2283|https://github.com/facebook/rocksdb/pull/2283]) Cons: * more work Given that we rarely upgrade the Rocks dependency, use only some fraction of its features, and would need or want to make only minimal changes ourselves, it seems like we could actually get away with very little extra work by forking rocksdb. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9147) zookeeper service not running
parimal created KAFKA-9147: -- Summary: zookeeper service not running Key: KAFKA-9147 URL: https://issues.apache.org/jira/browse/KAFKA-9147 Project: Kafka Issue Type: Test Affects Versions: 2.3.0 Environment: Ubuntu Reporter: parimal i was able to start zookeeper service on stand alone Ubuntu using the command root@N-5CG73531RZ:/# /usr/local/zookeeper/bin/zkServer.sh start /usr/bin/java ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED However when i do ps -ef I dont see any zookeeper service running root@N-5CG73531RZ:/# ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 Nov04 ? 00:00:00 /init root 5 1 0 Nov04 tty1 00:00:00 /init pgarg00 6 5 0 Nov04 tty1 00:00:00 -bash root 2861 6 0 Nov04 tty1 00:00:00 sudo -i root 2862 2861 0 Nov04 tty1 00:00:03 -bash root 5347 1 0 18:24 ? 00:00:00 /usr/sbin/sshd root 5367 1 0 18:25 ? 00:00:00 /usr/sbin/inetd root 8950 2862 0 19:15 tty1 00:00:00 ps -ef Also when I do telnet , connection is refused root@N-5CG73531RZ:/# telnet localhost 2181 Trying 127.0.0.1... telnet: Unable to connect to remote host: Connection refused can you plz help me ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8367) Non-heap memory leak in Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman resolved KAFKA-8367. Resolution: Fixed > Non-heap memory leak in Kafka Streams > - > > Key: KAFKA-8367 > URL: https://issues.apache.org/jira/browse/KAFKA-8367 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.2.0 >Reporter: Pavel Savov >Priority: Major > Attachments: memory-prod.png, memory-test.png > > > We have been observing a non-heap memory leak after upgrading to Kafka > Streams 2.2.0 from 2.0.1. We suspect the source to be around RocksDB as the > leak only happens when we enable stateful stream operations (utilizing > stores). We are aware of *KAFKA-8323* and have created our own fork of 2.2.0 > and ported the fix scheduled for release in 2.2.1 to our fork. It did not > stop the leak, however. > We are having this memory leak in our production environment where the > consumer group is auto-scaled in and out in response to changes in traffic > volume, and in our test environment where we have two consumers, no > autoscaling and relatively constant traffic. > Below is some information I'm hoping will be of help: > * RocksDB Config: > Block cache size: 4 MiB > Write buffer size: 2 MiB > Block size: 16 KiB > Cache index and filter blocks: true > Manifest preallocation size: 64 KiB > Max write buffer number: 3 > Max open files: 6144 > > * Memory usage in production > The attached graph (memory-prod.png) shows memory consumption for each > instance as a separate line. The horizontal red line at 6 GiB is the memory > limit. > As illustrated on the attached graph from production, memory consumption in > running instances goes up around autoscaling events (scaling the consumer > group either in or out) and associated rebalancing. It stabilizes until the > next autoscaling event but it never goes back down. > An example of scaling out can be seen from around 21:00 hrs where three new > instances are started in response to a traffic spike. > Just after midnight traffic drops and some instances are shut down. Memory > consumption in the remaining running instances goes up. > Memory consumption climbs again from around 6:00AM due to increased traffic > and new instances are being started until around 10:30AM. Memory consumption > never drops until the cluster is restarted around 12:30. > > * Memory usage in test > As illustrated by the attached graph (memory-test.png) we have a fixed number > of two instances in our test environment and no autoscaling. Memory > consumption rises linearly until it reaches the limit (around 2:00 AM on > 5/13) and Mesos restarts the offending instances, or we restart the cluster > manually. > > * No heap leaks observed > * Window retention: 2 or 11 minutes (depending on operation type) > * Issue not present in Kafka Streams 2.0.1 > * No memory leak for stateless stream operations (when no RocksDB stores are > used) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8367) Non-heap memory leak in Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967981#comment-16967981 ] Sophie Blee-Goldman commented on KAFKA-8367: [~pavelsavov] I am going to close this as the underlying issue seems to be KAFKA-8637. If you try the patch above or any of the version that includes this fix and still see a memory leak, please reopen this ticket and let us know > Non-heap memory leak in Kafka Streams > - > > Key: KAFKA-8367 > URL: https://issues.apache.org/jira/browse/KAFKA-8367 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.2.0 >Reporter: Pavel Savov >Priority: Major > Attachments: memory-prod.png, memory-test.png > > > We have been observing a non-heap memory leak after upgrading to Kafka > Streams 2.2.0 from 2.0.1. We suspect the source to be around RocksDB as the > leak only happens when we enable stateful stream operations (utilizing > stores). We are aware of *KAFKA-8323* and have created our own fork of 2.2.0 > and ported the fix scheduled for release in 2.2.1 to our fork. It did not > stop the leak, however. > We are having this memory leak in our production environment where the > consumer group is auto-scaled in and out in response to changes in traffic > volume, and in our test environment where we have two consumers, no > autoscaling and relatively constant traffic. > Below is some information I'm hoping will be of help: > * RocksDB Config: > Block cache size: 4 MiB > Write buffer size: 2 MiB > Block size: 16 KiB > Cache index and filter blocks: true > Manifest preallocation size: 64 KiB > Max write buffer number: 3 > Max open files: 6144 > > * Memory usage in production > The attached graph (memory-prod.png) shows memory consumption for each > instance as a separate line. The horizontal red line at 6 GiB is the memory > limit. > As illustrated on the attached graph from production, memory consumption in > running instances goes up around autoscaling events (scaling the consumer > group either in or out) and associated rebalancing. It stabilizes until the > next autoscaling event but it never goes back down. > An example of scaling out can be seen from around 21:00 hrs where three new > instances are started in response to a traffic spike. > Just after midnight traffic drops and some instances are shut down. Memory > consumption in the remaining running instances goes up. > Memory consumption climbs again from around 6:00AM due to increased traffic > and new instances are being started until around 10:30AM. Memory consumption > never drops until the cluster is restarted around 12:30. > > * Memory usage in test > As illustrated by the attached graph (memory-test.png) we have a fixed number > of two instances in our test environment and no autoscaling. Memory > consumption rises linearly until it reaches the limit (around 2:00 AM on > 5/13) and Mesos restarts the offending instances, or we restart the cluster > manually. > > * No heap leaks observed > * Window retention: 2 or 11 minutes (depending on operation type) > * Issue not present in Kafka Streams 2.0.1 > * No memory leak for stateless stream operations (when no RocksDB stores are > used) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9133) LogCleaner thread dies with: currentLog cannot be empty on an unexpected exception
[
https://issues.apache.org/jira/browse/KAFKA-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967976#comment-16967976
]
Karolis Pocius commented on KAFKA-9133:
---
Ended up downgrading to 2.3.0 and the issue went away. Seems like it was
introduced in KAFKA-8724
> LogCleaner thread dies with: currentLog cannot be empty on an unexpected
> exception
> --
>
> Key: KAFKA-9133
> URL: https://issues.apache.org/jira/browse/KAFKA-9133
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
>Affects Versions: 2.3.1
>Reporter: Karolis Pocius
>Priority: Major
>
> Log cleaner thread dies without a clear reference to which log is causing it:
> {code}
> [2019-11-02 11:59:59,078] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,144] INFO [kafka-log-cleaner-thread-0]: Starting
> (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,199] ERROR [kafka-log-cleaner-thread-0]: Error due to
> (kafka.log.LogCleaner)
> java.lang.IllegalStateException: currentLog cannot be empty on an unexpected
> exception
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:346)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:307)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
> Caused by: java.lang.IllegalArgumentException: Illegal request for non-active
> segments beginning at offset 5033130, which is larger than the active
> segment's base offset 5019648
> at kafka.log.Log.nonActiveLogSegmentsFrom(Log.scala:1933)
> at
> kafka.log.LogCleanerManager$.maxCompactionDelay(LogCleanerManager.scala:491)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$4(LogCleanerManager.scala:184)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.immutable.List.map(List.scala:298)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$1(LogCleanerManager.scala:181)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
> at
> kafka.log.LogCleanerManager.grabFilthiestCompactedLog(LogCleanerManager.scala:171)
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:321)
> ... 2 more
> [2019-11-02 11:59:59,200] INFO [kafka-log-cleaner-thread-0]: Stopped
> (kafka.log.LogCleaner)
> {code}
> If I try to ressurect it by dynamically bumping {{log.cleaner.threads}} it
> instantly dies with the exact same error.
> Not sure if this is something KAFKA-8725 is supposed to address.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9044) Brokers occasionally (randomly?) dropping out of clusters
[
https://issues.apache.org/jira/browse/KAFKA-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967975#comment-16967975
]
Peter Bukowinski commented on KAFKA-9044:
-
The broker doesn't actually shut down. It continues to run, but it believes
it's the only broker left. You can see in the log snippet above that after the
controller has kicked the broker out, the broker shrinks the ISR to only
contain itself.
> Brokers occasionally (randomly?) dropping out of clusters
> -
>
> Key: KAFKA-9044
> URL: https://issues.apache.org/jira/browse/KAFKA-9044
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.3.0, 2.3.1
> Environment: Ubuntu 14.04
>Reporter: Peter Bukowinski
>Priority: Major
>
> I have several cluster running kafka 2.3.1 and this issue has affected all of
> them. Because of replication and the size of the clusters (30 brokers), this
> bug is not causing any data loss, but it is nevertheless concerning. When a
> broker drops out, the log gives no indication that there are any zookeeper
> issues (and indeed the zookeepers are healthy when this occurs. Here's
> snippet from a broker log when it occurs:
> {{[2019-10-07 11:02:27,630] INFO [GroupMetadataManager brokerId=14] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Found deletable segments with base offsets [1975332] due to
> retention time 360ms breach (kafka.log.Log)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Scheduling log segment [baseOffset 1975332, size 92076008]
> for deletion. (kafka.log.Log)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Incrementing log start offset to 2000317 (kafka.log.Log)}}
> {{[2019-10-07 11:03:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Deleting segment 1975332 (kafka.log.Log)}}
> {{[2019-10-07 11:03:56,957] INFO Deleted log
> /data/3/kl/internal_test-52/01975332.log.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:03:56,957] INFO Deleted offset index
> /data/3/kl/internal_test-52/01975332.index.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:03:56,958] INFO Deleted time index
> /data/3/kl/internal_test-52/01975332.timeindex.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:12:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:42:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:52:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:02:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:12:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:42:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 1 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:52:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:02:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:12:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0
[jira] [Commented] (KAFKA-9044) Brokers occasionally (randomly?) dropping out of clusters
[
https://issues.apache.org/jira/browse/KAFKA-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967963#comment-16967963
]
Ismael Juma commented on KAFKA-9044:
Is it possible that something in the infrastructure is causing the broker to
shutdown?
> Brokers occasionally (randomly?) dropping out of clusters
> -
>
> Key: KAFKA-9044
> URL: https://issues.apache.org/jira/browse/KAFKA-9044
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.3.0, 2.3.1
> Environment: Ubuntu 14.04
>Reporter: Peter Bukowinski
>Priority: Major
>
> I have several cluster running kafka 2.3.1 and this issue has affected all of
> them. Because of replication and the size of the clusters (30 brokers), this
> bug is not causing any data loss, but it is nevertheless concerning. When a
> broker drops out, the log gives no indication that there are any zookeeper
> issues (and indeed the zookeepers are healthy when this occurs. Here's
> snippet from a broker log when it occurs:
> {{[2019-10-07 11:02:27,630] INFO [GroupMetadataManager brokerId=14] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Found deletable segments with base offsets [1975332] due to
> retention time 360ms breach (kafka.log.Log)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Scheduling log segment [baseOffset 1975332, size 92076008]
> for deletion. (kafka.log.Log)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Incrementing log start offset to 2000317 (kafka.log.Log)}}
> {{[2019-10-07 11:03:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Deleting segment 1975332 (kafka.log.Log)}}
> {{[2019-10-07 11:03:56,957] INFO Deleted log
> /data/3/kl/internal_test-52/01975332.log.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:03:56,957] INFO Deleted offset index
> /data/3/kl/internal_test-52/01975332.index.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:03:56,958] INFO Deleted time index
> /data/3/kl/internal_test-52/01975332.timeindex.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:12:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:42:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:52:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:02:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:12:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:42:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 1 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:52:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:02:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:12:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0
[jira] [Commented] (KAFKA-9142) topic __transaction_state disk space grows too big
[
https://issues.apache.org/jira/browse/KAFKA-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967944#comment-16967944
]
Francesca Capochiani commented on KAFKA-9142:
-
Same issue too!
> topic __transaction_state disk space grows too big
> --
>
> Key: KAFKA-9142
> URL: https://issues.apache.org/jira/browse/KAFKA-9142
> Project: Kafka
> Issue Type: Improvement
> Components: log cleaner
>Affects Versions: 2.0.0
>Reporter: panpan.liu
>Priority: Critical
> Attachments: image-2019-11-05-15-31-46-859.png,
> image-2019-11-05-15-33-55-632.png, image-2019-11-05-15-36-52-235.png
>
>
> kafka broker :2.0.0
> kafka stream client: 2.1.0
> topic description please see attachements.
> The broker runs from 10/2018. I find the topic grows up and does not delete
> (*cleanup.policy=compact*).
> How can I handle it ?
>
> {code:java}
> # topic description(default)
> Topic:__transaction_state PartitionCount:50 ReplicationFactor:2
> Configs:segment.bytes=104857600,unclean.leader.election.enable=false,compression.type=uncompressed,cleanup.policy=compact,min.insync.replicas=1
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9146) Add option to force delete members in stream reset tool
Boyang Chen created KAFKA-9146: -- Summary: Add option to force delete members in stream reset tool Key: KAFKA-9146 URL: https://issues.apache.org/jira/browse/KAFKA-9146 Project: Kafka Issue Type: Improvement Components: consumer, streams Reporter: Boyang Chen Sometimes people want to reset the stream application sooner, but blocked by the left-over members inside group coordinator, which only expire after session timeout. When user configures a really long session timeout, it could prevent the group from clearing. We should consider adding the support to cleanup members by forcing them to leave the group. To do that, # If the stream application is already on static membership, we could call directly from adminClient.removeMembersFromGroup # If the application is on dynamic membership, we should modify adminClient.removeMembersFromGroup interface to allow deletion based on member.id. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8769) Consider computing stream time independently per key
[
https://issues.apache.org/jira/browse/KAFKA-8769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967926#comment-16967926
]
Richard Yu commented on KAFKA-8769:
---
[~vvcephei] Do you have any ideas on advancing stream time? It seems unnatural
to do it artificially, so I wonder if we can just move ahead with the per key
stream time part of the issue, since that seems to be the meat of it.
> Consider computing stream time independently per key
>
>
> Key: KAFKA-8769
> URL: https://issues.apache.org/jira/browse/KAFKA-8769
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: John Roesler
>Priority: Major
> Labels: needs-discussion, needs-kip
>
> Currently, Streams uses a concept of "stream time", which is computed as the
> highest timestamp observed by stateful operators, per partition. This concept
> of time backs grace period, retention time, and suppression.
> For use cases in which data is produced to topics in roughly chronological
> order (as in db change capture), this reckoning is fine.
> Some use cases have a different pattern, though. For example, in IOT
> applications, it's common for sensors to save up quite a bit of data and then
> dump it all at once into the topic. See
> https://stackoverflow.com/questions/57082923/kafka-windowed-stream-make-grace-and-suppress-key-aware
> for a concrete example of the use case.
> I have heard of cases where each sensor dumps 24 hours' worth of data at a
> time into the topic. This results in a pattern in which, when reading a
> single partition, the operators observe a lot of consecutive records for one
> key that increase in timestamp for 24 hours, then a bunch of consecutive
> records for another key that are also increasing in timestamp over the same
> 24 hour period. With our current stream-time definition, this means that the
> partition's stream time increases while reading the first key's data, but
> then stays paused while reading the second key's data, since the second batch
> of records all have timestamps in the "past".
> E.g:
> {noformat}
> A@t0 (stream time: 0)
> A@t1 (stream time: 1)
> A@t2 (stream time: 2)
> A@t3 (stream time: 3)
> B@t0 (stream time: 3)
> B@t1 (stream time: 3)
> B@t2 (stream time: 3)
> B@t3 (stream time: 3)
> {noformat}
> This pattern results in an unfortunate compromise in which folks are required
> to set the grace period to the max expected time skew, for example 24 hours,
> or Streams will just drop the second key's data (since it is late). But, this
> means that if they want to use Suppression for "final results", they have to
> wait 24 hours for the result.
> This tradeoff is not strictly necessary, though, because each key represents
> a logically independent sequence of events. Tracking by partition is simply
> convenient, but typically not logically meaningful. That is, the partitions
> are just physically independent sequences of events, so it's convenient to
> track stream time at this granularity. It would be just as correct, and more
> useful for IOT-like use cases, to track time independently for each key.
> However, before considering this change, we need to solve the
> testing/low-traffic problem. This is the opposite issue, where a partition
> doesn't get enough traffic to advance stream time and results remain "stuck"
> in the suppression buffers. We can provide some mechanism to force the
> advancement of time across all partitions, for use in testing when you want
> to flush out all results, or in production when some topic is low volume. We
> shouldn't consider tracking time _more_ granularly until this problem is
> solved, since it would just make the low-traffic problem worse.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9044) Brokers occasionally (randomly?) dropping out of clusters
[
https://issues.apache.org/jira/browse/KAFKA-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967879#comment-16967879
]
Peter Bukowinski commented on KAFKA-9044:
-
I downgraded one of the affected clusters to version 2.3.0 and this issue is
still occurring on that cluster.
> Brokers occasionally (randomly?) dropping out of clusters
> -
>
> Key: KAFKA-9044
> URL: https://issues.apache.org/jira/browse/KAFKA-9044
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.3.0, 2.3.1
> Environment: Ubuntu 14.04
>Reporter: Peter Bukowinski
>Priority: Major
>
> I have several cluster running kafka 2.3.1 and this issue has affected all of
> them. Because of replication and the size of the clusters (30 brokers), this
> bug is not causing any data loss, but it is nevertheless concerning. When a
> broker drops out, the log gives no indication that there are any zookeeper
> issues (and indeed the zookeepers are healthy when this occurs. Here's
> snippet from a broker log when it occurs:
> {{[2019-10-07 11:02:27,630] INFO [GroupMetadataManager brokerId=14] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Found deletable segments with base offsets [1975332] due to
> retention time 360ms breach (kafka.log.Log)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Scheduling log segment [baseOffset 1975332, size 92076008]
> for deletion. (kafka.log.Log)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Incrementing log start offset to 2000317 (kafka.log.Log)}}
> {{[2019-10-07 11:03:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Deleting segment 1975332 (kafka.log.Log)}}
> {{[2019-10-07 11:03:56,957] INFO Deleted log
> /data/3/kl/internal_test-52/01975332.log.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:03:56,957] INFO Deleted offset index
> /data/3/kl/internal_test-52/01975332.index.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:03:56,958] INFO Deleted time index
> /data/3/kl/internal_test-52/01975332.timeindex.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:12:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:42:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:52:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:02:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:12:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:42:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 1 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:52:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:02:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:12:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
[jira] [Updated] (KAFKA-9044) Brokers occasionally (randomly?) dropping out of clusters
[
https://issues.apache.org/jira/browse/KAFKA-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bukowinski updated KAFKA-9044:
Affects Version/s: 2.3.0
> Brokers occasionally (randomly?) dropping out of clusters
> -
>
> Key: KAFKA-9044
> URL: https://issues.apache.org/jira/browse/KAFKA-9044
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.3.0, 2.3.1
> Environment: Ubuntu 14.04
>Reporter: Peter Bukowinski
>Priority: Major
>
> I have several cluster running kafka 2.3.1 and this issue has affected all of
> them. Because of replication and the size of the clusters (30 brokers), this
> bug is not causing any data loss, but it is nevertheless concerning. When a
> broker drops out, the log gives no indication that there are any zookeeper
> issues (and indeed the zookeepers are healthy when this occurs. Here's
> snippet from a broker log when it occurs:
> {{[2019-10-07 11:02:27,630] INFO [GroupMetadataManager brokerId=14] Removed 0
> expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Found deletable segments with base offsets [1975332] due to
> retention time 360ms breach (kafka.log.Log)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Scheduling log segment [baseOffset 1975332, size 92076008]
> for deletion. (kafka.log.Log)}}
> {{[2019-10-07 11:02:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Incrementing log start offset to 2000317 (kafka.log.Log)}}
> {{[2019-10-07 11:03:56,936] INFO [Log partition=internal_test-52,
> dir=/data/3/kl] Deleting segment 1975332 (kafka.log.Log)}}
> {{[2019-10-07 11:03:56,957] INFO Deleted log
> /data/3/kl/internal_test-52/01975332.log.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:03:56,957] INFO Deleted offset index
> /data/3/kl/internal_test-52/01975332.index.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:03:56,958] INFO Deleted time index
> /data/3/kl/internal_test-52/01975332.timeindex.deleted.
> (kafka.log.LogSegment)}}
> {{[2019-10-07 11:12:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:42:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 11:52:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:02:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:12:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:42:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 1 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 12:52:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:02:27,630] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:12:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:22:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:32:27,629] INFO [GroupMetadataManager brokerId=14] Removed
> 0 expired offsets in 0 milliseconds.
> (kafka.coordinator.group.GroupMetadataManager)}}
> {{[2019-10-07 13:42:27,629]
[jira] [Created] (KAFKA-9145) AbstractCoordinator should respect retry backoff between rebalances
Jason Gustafson created KAFKA-9145: -- Summary: AbstractCoordinator should respect retry backoff between rebalances Key: KAFKA-9145 URL: https://issues.apache.org/jira/browse/KAFKA-9145 Project: Kafka Issue Type: Improvement Components: consumer, KafkaConnect Reporter: Jason Gustafson Fix For: 2.5.0 We hit a situation where the connect worker seems to have been requesting group rejoins repeatedly for some reason. In theory the same thing could happen to the consumer if a user repeatedly changes the subscribed topics in a tight loop. Currently AbstractCoordinator would not backoff in these cases because the rebalances may all be completing successfully. It probably makes sense to enforce a minimum time between locally triggered rebalances in order to avoid overloading the broker with repeated group rebalances. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-7061) Enhanced log compaction
[ https://issues.apache.org/jira/browse/KAFKA-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Senthilnathan Muthusamy updated KAFKA-7061: --- Description: Enhance log compaction to support more than just offset comparison, so the insertion order isn't dictating which records to keep. Default behavior is kept as it was, with the enhanced approached having to be purposely activated. The enhanced compaction is done either via the record timestamp, by settings the new configuration as "timestamp" or via the record headers by setting this configuration to anything other than the default "offset" or the reserved "timestamp". See [KIP-280|https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction] for more details. +From Guozhang:+ We should emphasize on the WIKI that the newly introduced config yields to the existing "log.cleanup.policy", i.e. if the latter's value is `delete` not `compact`, then the previous config would be ignored. was: Enhance log compaction to support more than just offset comparison, so the insertion order isn't dictating which records to keep. Default behavior is kept as it was, with the enhanced approached having to be purposely activated. The enhanced compaction is done either via the record timestamp, by settings the new configuration as "timestamp" or via the record headers by setting this configuration to anything other than the default "offset" or the reserved "timestamp". See [KIP-280|https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction] for more details. +From Guozhang:+ We should emphasize that the newly introduced config yields to the existing "log.cleanup.policy", i.e. if the latter's value is `delete` not `compact`, then the previous config would be ignored. > Enhanced log compaction > --- > > Key: KAFKA-7061 > URL: https://issues.apache.org/jira/browse/KAFKA-7061 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 2.5.0 >Reporter: Luis Cabral >Assignee: Senthilnathan Muthusamy >Priority: Major > Labels: kip > > Enhance log compaction to support more than just offset comparison, so the > insertion order isn't dictating which records to keep. > Default behavior is kept as it was, with the enhanced approached having to be > purposely activated. > The enhanced compaction is done either via the record timestamp, by settings > the new configuration as "timestamp" or via the record headers by setting > this configuration to anything other than the default "offset" or the > reserved "timestamp". > See > [KIP-280|https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction] > for more details. > +From Guozhang:+ We should emphasize on the WIKI that the newly introduced > config yields to the existing "log.cleanup.policy", i.e. if the latter's > value is `delete` not `compact`, then the previous config would be ignored. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-7061) Enhanced log compaction
[ https://issues.apache.org/jira/browse/KAFKA-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Senthilnathan Muthusamy updated KAFKA-7061: --- Description: Enhance log compaction to support more than just offset comparison, so the insertion order isn't dictating which records to keep. Default behavior is kept as it was, with the enhanced approached having to be purposely activated. The enhanced compaction is done either via the record timestamp, by settings the new configuration as "timestamp" or via the record headers by setting this configuration to anything other than the default "offset" or the reserved "timestamp". See [KIP-280|https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction] for more details. +From Guozhang:+ We should emphasize that the newly introduced config yields to the existing "log.cleanup.policy", i.e. if the latter's value is `delete` not `compact`, then the previous config would be ignored. was: Enhance log compaction to support more than just offset comparison, so the insertion order isn't dictating which records to keep. Default behavior is kept as it was, with the enhanced approached having to be purposely activated. The enhanced compaction is done either via the record timestamp, by settings the new configuration as "timestamp" or via the record headers by setting this configuration to anything other than the default "offset" or the reserved "timestamp". See [KIP-280|https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction] for more details. > Enhanced log compaction > --- > > Key: KAFKA-7061 > URL: https://issues.apache.org/jira/browse/KAFKA-7061 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 2.5.0 >Reporter: Luis Cabral >Assignee: Senthilnathan Muthusamy >Priority: Major > Labels: kip > > Enhance log compaction to support more than just offset comparison, so the > insertion order isn't dictating which records to keep. > Default behavior is kept as it was, with the enhanced approached having to be > purposely activated. > The enhanced compaction is done either via the record timestamp, by settings > the new configuration as "timestamp" or via the record headers by setting > this configuration to anything other than the default "offset" or the > reserved "timestamp". > See > [KIP-280|https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction] > for more details. > +From Guozhang:+ We should emphasize that the newly introduced config yields > to the existing "log.cleanup.policy", i.e. if the latter's value is `delete` > not `compact`, then the previous config would be ignored. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7987) a broker's ZK session may die on transient auth failure
[
https://issues.apache.org/jira/browse/KAFKA-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967806#comment-16967806
]
Jun Rao commented on KAFKA-7987:
[~gcampbell], we could try calling scheduleSessionExpiryHandler() in
ZooKeeperClientWatcher.process() when authentication fails. We have to think a
bit more whether we still need StateChangeHandler.onAuthFailure() or not.
> a broker's ZK session may die on transient auth failure
> ---
>
> Key: KAFKA-7987
> URL: https://issues.apache.org/jira/browse/KAFKA-7987
> Project: Kafka
> Issue Type: Improvement
>Reporter: Jun Rao
>Priority: Major
>
> After a transient network issue, we saw the following log in a broker.
> {code:java}
> [23:37:02,102] ERROR SASL authentication with Zookeeper Quorum member failed:
> javax.security.sasl.SaslException: An error:
> (java.security.PrivilegedActionException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Server not found in Kerberos database (7))]) occurred when
> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client
> will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
> [23:37:02,102] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> {code}
> The network issue prevented the broker from communicating to ZK. The broker's
> ZK session then expired, but the broker didn't know that yet since it
> couldn't establish a connection to ZK. When the network was back, the broker
> tried to establish a connection to ZK, but failed due to auth failure (likely
> due to a transient KDC issue). The current logic just ignores the auth
> failure without trying to create a new ZK session. Then the broker will be
> permanently in a state that it's alive, but not registered in ZK.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-7987) a broker's ZK session may die on transient auth failure
[
https://issues.apache.org/jira/browse/KAFKA-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967790#comment-16967790
]
Graham Campbell commented on KAFKA-7987:
[~junrao] We're starting to see these errors more frequently (guess our network
is getting less reliable), so I'm looking at a fix for this. Does scheduling a
reinitialize() in the ZookeeperClient after notifying handlers seem like a
reasonable solution? I don't see a way to get more details from the ZK client
to try to tell if the auth failure was caused by a retriable error or not.
> a broker's ZK session may die on transient auth failure
> ---
>
> Key: KAFKA-7987
> URL: https://issues.apache.org/jira/browse/KAFKA-7987
> Project: Kafka
> Issue Type: Improvement
>Reporter: Jun Rao
>Priority: Major
>
> After a transient network issue, we saw the following log in a broker.
> {code:java}
> [23:37:02,102] ERROR SASL authentication with Zookeeper Quorum member failed:
> javax.security.sasl.SaslException: An error:
> (java.security.PrivilegedActionException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Server not found in Kerberos database (7))]) occurred when
> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client
> will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
> [23:37:02,102] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> {code}
> The network issue prevented the broker from communicating to ZK. The broker's
> ZK session then expired, but the broker didn't know that yet since it
> couldn't establish a connection to ZK. When the network was back, the broker
> tried to establish a connection to ZK, but failed due to auth failure (likely
> due to a transient KDC issue). The current logic just ignores the auth
> failure without trying to create a new ZK session. Then the broker will be
> permanently in a state that it's alive, but not registered in ZK.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9144) Early expiration of producer state can cause coordinator epoch to regress
Jason Gustafson created KAFKA-9144: -- Summary: Early expiration of producer state can cause coordinator epoch to regress Key: KAFKA-9144 URL: https://issues.apache.org/jira/browse/KAFKA-9144 Project: Kafka Issue Type: Bug Reporter: Jason Gustafson Assignee: Jason Gustafson Transaction markers are written by the transaction coordinator. In order to fence zombie coordinators, we use the leader epoch associated with the coordinator partition. Partition leaders verify the epoch in the WriteTxnMarker request and ensure that it can only increase. However, when producer state expires, we stop tracking the epoch and it is possible for monotonicity to be violated. Generally we expect expiration to be on the order of days, so it should be unlikely for this to be a problem. At least that is the theory. We observed a case where a coordinator epoch decreased between nearly consecutive writes within a couple minutes of each other. Upon investigation, we found that producer state had been incorrectly expired. We believe the sequence of events is the following: # Producer writes transactional data and fails before committing # Coordinator times out the transaction and writes ABORT markers # Upon seeing the ABORT and the bumped epoch, the partition leader deletes state from the last epoch, which effectively resets the last timestamp for the producer to -1. # The coordinator becomes a zombie before getting a successful response and continues trying to send # The new coordinator notices the incomplete transaction and also sends markers # The partition leader accepts the write from the new coordinator # The producer state is expired because the last timestamp was -1 # The partition leader accepts the write from the old coordinator Basically it takes an alignment of planets to hit this bug, but it is possible. If you hit it, then the broker may be unable to start because we validate epoch monotonicity during log recovery. The problem is in 3 when the timestamp gets reset. We should use the timestamp from the marker instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8977) Remove MockStreamsMetrics Since it is not a Mock
[
https://issues.apache.org/jira/browse/KAFKA-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967683#comment-16967683
]
bibin sebastian commented on KAFKA-8977:
[~cadonna] do you think that my PR above is up for review?
> Remove MockStreamsMetrics Since it is not a Mock
>
>
> Key: KAFKA-8977
> URL: https://issues.apache.org/jira/browse/KAFKA-8977
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Bruno Cadonna
>Assignee: bibin sebastian
>Priority: Minor
> Labels: newbie
>
> The class {{MockStreamsMetrics}} is used throughout unit tests as a mock but
> it is not really a mock since it only hides two parameters of the
> {{StreamsMetricsImpl}} constructor. Either a real mock or the real
> {{StreamsMetricsImpl}} should be used in the tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9143) DistributedHerder misleadingly log error on connector task reconfiguration
[
https://issues.apache.org/jira/browse/KAFKA-9143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967661#comment-16967661
]
ASF GitHub Bot commented on KAFKA-9143:
---
ivanyu commented on pull request #7648: KAFKA-9143: Log task reconfiguration
error only when it happened
URL: https://github.com/apache/kafka/pull/7648
This commit makes `DistributedHerder` log that some error has happened
during task reconfiguration only when it actually has happened.
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DistributedHerder misleadingly log error on connector task reconfiguration
> --
>
> Key: KAFKA-9143
> URL: https://issues.apache.org/jira/browse/KAFKA-9143
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Ivan Yurchenko
>Assignee: Ivan Yurchenko
>Priority: Minor
>
> In {{DistributedHerder}} in {{reconfigureConnectorTasksWithRetry}} method
> there's a
> [callback|https://github.com/apache/kafka/blob/c552c06aed50b4d4d9a85f73ccc89bc06fa7e094/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1247]:
> {code:java}
> @Override
> public void onCompletion(Throwable error, Void result) {
> log.error("Unexpected error during connector task reconfiguration: ",
> error);
> log.error("Task reconfiguration for {} failed unexpectedly, this
> connector will not be properly reconfigured unless manually triggered.",
> connName);
> }
> {code}
> It an error message even when the operation succeeded (i.e., {{error}} is
> {{null}}).
> It should include {{if (error != null)}} condition, like in the same class
> [in another
> method|https://github.com/apache/kafka/blob/c552c06aed50b4d4d9a85f73ccc89bc06fa7e094/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L792].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9143) DistributedHerder misleadingly log error on connector task reconfiguration
Ivan Yurchenko created KAFKA-9143:
-
Summary: DistributedHerder misleadingly log error on connector
task reconfiguration
Key: KAFKA-9143
URL: https://issues.apache.org/jira/browse/KAFKA-9143
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Ivan Yurchenko
Assignee: Ivan Yurchenko
In {{DistributedHerder}} in {{reconfigureConnectorTasksWithRetry}} method
there's a
[callback|https://github.com/apache/kafka/blob/c552c06aed50b4d4d9a85f73ccc89bc06fa7e094/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1247]:
{code:java}
@Override
public void onCompletion(Throwable error, Void result) {
log.error("Unexpected error during connector task reconfiguration: ",
error);
log.error("Task reconfiguration for {} failed unexpectedly, this connector
will not be properly reconfigured unless manually triggered.", connName);
}
{code}
It an error message even when the operation succeeded (i.e., {{error}} is
{{null}}).
It should include {{if (error != null)}} condition, like in the same class [in
another
method|https://github.com/apache/kafka/blob/c552c06aed50b4d4d9a85f73ccc89bc06fa7e094/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L792].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-9133) LogCleaner thread dies with: currentLog cannot be empty on an unexpected exception
[
https://issues.apache.org/jira/browse/KAFKA-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967572#comment-16967572
]
Tim Van Laer edited comment on KAFKA-9133 at 11/5/19 2:37 PM:
--
[~Karolis] and i did a couple of code changes in an attempt to workaround this
issue, so far we didn't succeed.
It looks like this issue always appear on topics that have
{{cleanup.policy=compact,delete}}.
We tried to patch:
* kafka.log.LogCleanerManager#cleanableOffsets: to return logStartOffset in
case the checkpointDirtyOffset is bigger than the activeSegment.baseOffset
* kafka.log.LogCleanerManager#cleanableOffsets: to return
activeSegment.baseOffset in case the checkpointDirtyOffset is bigger than the
activeSegment.baseOffset
* kafka.log.Log#nonActiveLogSegmentsFrom: in case from >
activeSegment.baseOffset, return empty collection instead of throwing exception
--> this keeps the cleaner thread running but the same partitions are picked as
filthiest over and over, so this basically doesn't help. I didn't investigate
if I can tweak the filthiest log selection.
I'm confused on how the log start could be bigger than the active segment
start. The only thing I can currently think of is some race condition between
the merging of older small compacted segments into one and the actual
compaction but as far as i understand he code it's all on the same thread.
was (Author: timvanlaer):
[~Karolis] and i did a couple of code changes in an attempt to workaround this
issue, so far we didn't succeed.
It looks like this issue always appear on topics that have
`cleanup.policy=compact,delete`.
We tried to patch:
* kafka.log.LogCleanerManager#cleanableOffsets: to return logStartOffset in
case the checkpointDirtyOffset is bigger than the activeSegment.baseOffset
* kafka.log.LogCleanerManager#cleanableOffsets: to return
activeSegment.baseOffset in case the checkpointDirtyOffset is bigger than the
activeSegment.baseOffset
* kafka.log.Log#nonActiveLogSegmentsFrom: in case from >
activeSegment.baseOffset, return empty collection instead of throwing exception
--> this keeps the cleaner thread running but the same partitions are picked as
filthiest over and over, so this basically doesn't help. I didn't investigate
if I can tweak the filthiest log selection.
I'm confused on how the log start could be bigger than the active segment
start. The only thing I can currently think of is some race condition between
the merging of older small compacted segments into one and the actual
compaction but as far as i understand he code it's all on the same thread.
> LogCleaner thread dies with: currentLog cannot be empty on an unexpected
> exception
> --
>
> Key: KAFKA-9133
> URL: https://issues.apache.org/jira/browse/KAFKA-9133
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
>Affects Versions: 2.3.1
>Reporter: Karolis Pocius
>Priority: Major
>
> Log cleaner thread dies without a clear reference to which log is causing it:
> {code}
> [2019-11-02 11:59:59,078] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,144] INFO [kafka-log-cleaner-thread-0]: Starting
> (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,199] ERROR [kafka-log-cleaner-thread-0]: Error due to
> (kafka.log.LogCleaner)
> java.lang.IllegalStateException: currentLog cannot be empty on an unexpected
> exception
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:346)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:307)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
> Caused by: java.lang.IllegalArgumentException: Illegal request for non-active
> segments beginning at offset 5033130, which is larger than the active
> segment's base offset 5019648
> at kafka.log.Log.nonActiveLogSegmentsFrom(Log.scala:1933)
> at
> kafka.log.LogCleanerManager$.maxCompactionDelay(LogCleanerManager.scala:491)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$4(LogCleanerManager.scala:184)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.immutable.List.map(List.scala:298)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$1(LogCleanerManager.scala:181)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
> at
> kafka.log.LogCleanerManager.grabFilthiestCompactedLog(LogCleanerManager.scala:171)
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:321)
> ... 2 more
> [2019-11-02 11:59:59,200]
[jira] [Commented] (KAFKA-9133) LogCleaner thread dies with: currentLog cannot be empty on an unexpected exception
[
https://issues.apache.org/jira/browse/KAFKA-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967572#comment-16967572
]
Tim Van Laer commented on KAFKA-9133:
-
[~Karolis] and i did a couple of code changes in an attempt to workaround this
issue, so far we didn't succeed.
It looks like this issue always appear on topics that have
`cleanup.policy=compact,delete`.
We tried to patch:
* kafka.log.LogCleanerManager#cleanableOffsets: to return logStartOffset in
case the checkpointDirtyOffset is bigger than the activeSegment.baseOffset
* kafka.log.LogCleanerManager#cleanableOffsets: to return
activeSegment.baseOffset in case the checkpointDirtyOffset is bigger than the
activeSegment.baseOffset
* kafka.log.Log#nonActiveLogSegmentsFrom: in case from >
activeSegment.baseOffset, return empty collection instead of throwing exception
--> this keeps the cleaner thread running but the same partitions are picked as
filthiest over and over, so this basically doesn't help. I didn't investigate
if I can tweak the filthiest log selection.
I'm confused on how the log start could be bigger than the active segment
start. The only thing I can currently think of is some race condition between
the merging of older small compacted segments into one and the actual
compaction but as far as i understand he code it's all on the same thread.
> LogCleaner thread dies with: currentLog cannot be empty on an unexpected
> exception
> --
>
> Key: KAFKA-9133
> URL: https://issues.apache.org/jira/browse/KAFKA-9133
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
>Affects Versions: 2.3.1
>Reporter: Karolis Pocius
>Priority: Major
>
> Log cleaner thread dies without a clear reference to which log is causing it:
> {code}
> [2019-11-02 11:59:59,078] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,144] INFO [kafka-log-cleaner-thread-0]: Starting
> (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,199] ERROR [kafka-log-cleaner-thread-0]: Error due to
> (kafka.log.LogCleaner)
> java.lang.IllegalStateException: currentLog cannot be empty on an unexpected
> exception
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:346)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:307)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
> Caused by: java.lang.IllegalArgumentException: Illegal request for non-active
> segments beginning at offset 5033130, which is larger than the active
> segment's base offset 5019648
> at kafka.log.Log.nonActiveLogSegmentsFrom(Log.scala:1933)
> at
> kafka.log.LogCleanerManager$.maxCompactionDelay(LogCleanerManager.scala:491)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$4(LogCleanerManager.scala:184)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.immutable.List.map(List.scala:298)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$1(LogCleanerManager.scala:181)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
> at
> kafka.log.LogCleanerManager.grabFilthiestCompactedLog(LogCleanerManager.scala:171)
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:321)
> ... 2 more
> [2019-11-02 11:59:59,200] INFO [kafka-log-cleaner-thread-0]: Stopped
> (kafka.log.LogCleaner)
> {code}
> If I try to ressurect it by dynamically bumping {{log.cleaner.threads}} it
> instantly dies with the exact same error.
> Not sure if this is something KAFKA-8725 is supposed to address.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9110) Improve efficiency of disk reads when TLS is enabled
[ https://issues.apache.org/jira/browse/KAFKA-9110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967499#comment-16967499 ] ASF GitHub Bot commented on KAFKA-9110: --- ijuma commented on pull request #7604: KAFKA-9110: Improve efficiency of disk reads when TLS is enabled URL: https://github.com/apache/kafka/pull/7604 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve efficiency of disk reads when TLS is enabled > > > Key: KAFKA-9110 > URL: https://issues.apache.org/jira/browse/KAFKA-9110 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > Fix For: 2.5.0 > > > We currently do 8k reads and do unnecessary copies and allocations in every > read. Increasing the read size is particularly helpful for magnetic disks and > avoiding the copies and allocations improves CPU efficiency. > See the pull request for more details: > [https://github.com/apache/kafka/pull/7604] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9133) LogCleaner thread dies with: currentLog cannot be empty on an unexpected exception
[
https://issues.apache.org/jira/browse/KAFKA-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967451#comment-16967451
]
Karolis Pocius commented on KAFKA-9133:
---
Combing through the logs of the latest instance that crashed, I noticed the
following.
server.log reports this:
{code}
[2019-11-05 09:58:41,093] INFO [Log partition=xyz-state.0-changelog-10,
dir=/kafka-data] Incrementing log start offset to 56778 (kafka.log.Log)
{code}
and then a few seconds later log-cleaner.log:
{code}
[2019-11-05 09:58:50,168] ERROR [kafka-log-cleaner-thread-0]: Error due to
(kafka.log.LogCleaner)
java.lang.IllegalStateException: currentLog cannot be empty on an unexpected
exception
at
kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:346)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:307)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
Caused by: java.lang.IllegalArgumentException: Illegal request for non-active
segments beginning at offset 56778, which is larger than the active segment's
base offset 56642
at kafka.log.Log.nonActiveLogSegmentsFrom(Log.scala:1933)
at
kafka.log.LogCleanerManager$.maxCompactionDelay(LogCleanerManager.scala:491)
at
kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$4(LogCleanerManager.scala:184)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike.map(TraversableLike.scala:238)
at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
at scala.collection.immutable.List.map(List.scala:298)
at
kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$1(LogCleanerManager.scala:181)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at
kafka.log.LogCleanerManager.grabFilthiestCompactedLog(LogCleanerManager.scala:171)
at
kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:321)
... 2 more
[2019-11-05 09:58:50,170] INFO [kafka-log-cleaner-thread-0]: Stopped
(kafka.log.LogCleaner)
{code}
> LogCleaner thread dies with: currentLog cannot be empty on an unexpected
> exception
> --
>
> Key: KAFKA-9133
> URL: https://issues.apache.org/jira/browse/KAFKA-9133
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
>Affects Versions: 2.3.1
>Reporter: Karolis Pocius
>Priority: Major
>
> Log cleaner thread dies without a clear reference to which log is causing it:
> {code}
> [2019-11-02 11:59:59,078] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,144] INFO [kafka-log-cleaner-thread-0]: Starting
> (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,199] ERROR [kafka-log-cleaner-thread-0]: Error due to
> (kafka.log.LogCleaner)
> java.lang.IllegalStateException: currentLog cannot be empty on an unexpected
> exception
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:346)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:307)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
> Caused by: java.lang.IllegalArgumentException: Illegal request for non-active
> segments beginning at offset 5033130, which is larger than the active
> segment's base offset 5019648
> at kafka.log.Log.nonActiveLogSegmentsFrom(Log.scala:1933)
> at
> kafka.log.LogCleanerManager$.maxCompactionDelay(LogCleanerManager.scala:491)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$4(LogCleanerManager.scala:184)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.immutable.List.map(List.scala:298)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$1(LogCleanerManager.scala:181)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
> at
> kafka.log.LogCleanerManager.grabFilthiestCompactedLog(LogCleanerManager.scala:171)
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:321)
> ... 2 more
> [2019-11-02 11:59:59,200] INFO [kafka-log-cleaner-thread-0]: Stopped
> (kafka.log.LogCleaner)
> {code}
> If I try to ressurect it by dynamically bumping {{log.cleaner.threads}} it
> instantly dies with the exact same error.
> Not sure if this is something KAFKA-8725 is supposed to address.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
