[jira] [Commented] (KAFKA-14069) Allow custom configuration of foreign key join internal topics
[ https://issues.apache.org/jira/browse/KAFKA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566714#comment-17566714 ] Emmanuel Brard commented on KAFKA-14069: I think it might be a bug I don't see any Delete requests being processed from the Kafka cluster log for topic with this pattern "-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-". > Allow custom configuration of foreign key join internal topics > -- > > Key: KAFKA-14069 > URL: https://issues.apache.org/jira/browse/KAFKA-14069 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.8.0 >Reporter: Emmanuel Brard >Priority: Minor > > Internal topic supporting foreign key joins (-subscription-registration-topic > and -subscription-response-topic) are automatically created with_ infinite > retention_ (retention.ms=-1, retention.bytes=-1). > As far as I understand those topics are used for communication between tasks > that are involved in the FK, the intermediate result though is persisted in a > compacted topic (-subscription-store-changelog). > This means, if I understood right, that during normal operation of the stream > application, once a message is read from the registration/subscription topic, > it will not be read again, even in case of recovery (the position in those > topics is committed). > Because we have very large tables being joined this way with very high > changes frequency, we end up with FK internal topics in the order of 1 or 2 > TB. This is complicated to maintain especially in term of disk space. > I was wondering if: > - this infinite retention is really a required configuration and if not > - this infinite retention could be replaced with a configurable one (for > example of 1 week, meaning that I accept that in case of failure I must this > my app within one week) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14069) Allow custom configuration of foreign key join internal topics
[ https://issues.apache.org/jira/browse/KAFKA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566695#comment-17566695 ] Emmanuel Brard commented on KAFKA-14069: One thing I did not mention yet is that we use version 2.8.0. I guess those API calls are triggered by the streaming application itself, should we run it in debug log mode to see the calls? > Allow custom configuration of foreign key join internal topics > -- > > Key: KAFKA-14069 > URL: https://issues.apache.org/jira/browse/KAFKA-14069 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.8.0 >Reporter: Emmanuel Brard >Priority: Minor > > Internal topic supporting foreign key joins (-subscription-registration-topic > and -subscription-response-topic) are automatically created with_ infinite > retention_ (retention.ms=-1, retention.bytes=-1). > As far as I understand those topics are used for communication between tasks > that are involved in the FK, the intermediate result though is persisted in a > compacted topic (-subscription-store-changelog). > This means, if I understood right, that during normal operation of the stream > application, once a message is read from the registration/subscription topic, > it will not be read again, even in case of recovery (the position in those > topics is committed). > Because we have very large tables being joined this way with very high > changes frequency, we end up with FK internal topics in the order of 1 or 2 > TB. This is complicated to maintain especially in term of disk space. > I was wondering if: > - this infinite retention is really a required configuration and if not > - this infinite retention could be replaced with a configurable one (for > example of 1 week, meaning that I accept that in case of failure I must this > my app within one week) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14069) Allow custom configuration of foreign key join internal topics
[ https://issues.apache.org/jira/browse/KAFKA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emmanuel Brard updated KAFKA-14069: --- Affects Version/s: 2.8.0 > Allow custom configuration of foreign key join internal topics > -- > > Key: KAFKA-14069 > URL: https://issues.apache.org/jira/browse/KAFKA-14069 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.8.0 >Reporter: Emmanuel Brard >Priority: Minor > > Internal topic supporting foreign key joins (-subscription-registration-topic > and -subscription-response-topic) are automatically created with_ infinite > retention_ (retention.ms=-1, retention.bytes=-1). > As far as I understand those topics are used for communication between tasks > that are involved in the FK, the intermediate result though is persisted in a > compacted topic (-subscription-store-changelog). > This means, if I understood right, that during normal operation of the stream > application, once a message is read from the registration/subscription topic, > it will not be read again, even in case of recovery (the position in those > topics is committed). > Because we have very large tables being joined this way with very high > changes frequency, we end up with FK internal topics in the order of 1 or 2 > TB. This is complicated to maintain especially in term of disk space. > I was wondering if: > - this infinite retention is really a required configuration and if not > - this infinite retention could be replaced with a configurable one (for > example of 1 week, meaning that I accept that in case of failure I must this > my app within one week) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-14069) Allow custom configuration of foreign key join internal topics
Emmanuel Brard created KAFKA-14069: -- Summary: Allow custom configuration of foreign key join internal topics Key: KAFKA-14069 URL: https://issues.apache.org/jira/browse/KAFKA-14069 Project: Kafka Issue Type: Improvement Components: streams Reporter: Emmanuel Brard Internal topic supporting foreign key joins (-subscription-registration-topic and -subscription-response-topic) are automatically created with_ infinite retention_ (retention.ms=-1, retention.bytes=-1). As far as I understand those topics are used for communication between tasks that are involved in the FK, the intermediate result though is persisted in a compacted topic (-subscription-store-changelog). This means, if I understood right, that during normal operation of the stream application, once a message is read from the registration/subscription topic, it will not be read again, even in case of recovery (the position in those topics is committed). Because we have very large tables being joined this way with very high changes frequency, we end up with FK internal topics in the order of 1 or 2 TB. This is complicated to maintain especially in term of disk space. I was wondering if: - this infinite retention is really a required configuration and if not - this infinite retention could be replaced with a configurable one (for example of 1 week, meaning that I accept that in case of failure I must this my app within one week) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-6951) Implement offset expiration semantics for unsubscribed topics
[
https://issues.apache.org/jira/browse/KAFKA-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820132#comment-16820132
]
Emmanuel Brard commented on KAFKA-6951:
---
[~wushujames] it's not implemented yet:
{code:java}
root@kafka-1:/# kafka-consumer-groups --bootstrap-server localhost:9092
--describe --group my_favorite_group
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST
CLIENT-ID
bar 0 58 58 0
kafka-python-1.4.6-5f66aa52-8497-404c-9c3e-69f36139b30d /172.19.0.5
kafka-python-1.4.6
foo 0 56 58 2 -
- -
{code}
{code:java}
root@kafka-1:/# kafka-consumer-groups --bootstrap-server localhost:9091 --group
my_favorite_group --topic foo --delete
The consumer does not support topic-specific offset deletion from a consumer
group.
{code}
See
https://github.com/apache/kafka/blob/47a9871ef65a13f9d58d5ea216de340f7e123da5/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala#L941
> Implement offset expiration semantics for unsubscribed topics
> -
>
> Key: KAFKA-6951
> URL: https://issues.apache.org/jira/browse/KAFKA-6951
> Project: Kafka
> Issue Type: Improvement
> Components: core
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Major
>
> [This
> portion|https://cwiki.apache.org/confluence/display/KAFKA/KIP-211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets#KIP-211:ReviseExpirationSemanticsofConsumerGroupOffsets-UnsubscribingfromaTopic]
> of KIP-211 will be implemented separately from the main PR.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
