[jira] [Commented] (KAFKA-4587) Rethink Unification of Caching with Dedupping
[
https://issues.apache.org/jira/browse/KAFKA-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171088#comment-17171088
]
Matthias J. Sax commented on KAFKA-4587:
In general yes, I guess. Why?
> Rethink Unification of Caching with Dedupping
> -
>
> Key: KAFKA-4587
> URL: https://issues.apache.org/jira/browse/KAFKA-4587
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Priority: Major
>
> This is discussed in PR https://github.com/apache/kafka/pull/1588
> In order to support user-customizable state store suppliers in the DSL we did
> the following:
> 1) Introduce a {{TupleForwarder}} to forward tuples from cached stores that
> is wrapping user customized stores.
> 2) Narrow the scope to only dedup on forwarding if it is the default
> CachingXXStore with wrapper RocksDB.
> With this, the unification of dedupping and caching is less useful now, and
> we are complicating the inner implementations for forwarding a lot. We need
> to re-consider this feature with finer granularity of turning on / off
> caching per store, potentially with explicit triggers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-4587) Rethink Unification of Caching with Dedupping
[
https://issues.apache.org/jira/browse/KAFKA-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170971#comment-17170971
]
Boyang Chen commented on KAFKA-4587:
What's the status of this ticket? Do we still want to work on it? [~guozhang]
[~mjsax]
> Rethink Unification of Caching with Dedupping
> -
>
> Key: KAFKA-4587
> URL: https://issues.apache.org/jira/browse/KAFKA-4587
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Priority: Major
>
> This is discussed in PR https://github.com/apache/kafka/pull/1588
> In order to support user-customizable state store suppliers in the DSL we did
> the following:
> 1) Introduce a {{TupleForwarder}} to forward tuples from cached stores that
> is wrapping user customized stores.
> 2) Narrow the scope to only dedup on forwarding if it is the default
> CachingXXStore with wrapper RocksDB.
> With this, the unification of dedupping and caching is less useful now, and
> we are complicating the inner implementations for forwarding a lot. We need
> to re-consider this feature with finer granularity of turning on / off
> caching per store, potentially with explicit triggers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-4587) Rethink Unification of Caching with Dedupping
[
https://issues.apache.org/jira/browse/KAFKA-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069296#comment-17069296
]
Matthias J. Sax commented on KAFKA-4587:
It is still relevant. In 2.1 release we added `suppress()` operator (cf
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables])
that is one step into this direction but caching and forwarding is still
coupled in the DSL.
Overall, for cleaner semantics it would be good to decouple caching for local
stores (and writing into the changelog topics) from downstream rate control as
we have `suppress()` for downstream rate control now.
> Rethink Unification of Caching with Dedupping
> -
>
> Key: KAFKA-4587
> URL: https://issues.apache.org/jira/browse/KAFKA-4587
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Priority: Major
>
> This is discussed in PR https://github.com/apache/kafka/pull/1588
> In order to support user-customizable state store suppliers in the DSL we did
> the following:
> 1) Introduce a {{TupleForwarder}} to forward tuples from cached stores that
> is wrapping user customized stores.
> 2) Narrow the scope to only dedup on forwarding if it is the default
> CachingXXStore with wrapper RocksDB.
> With this, the unification of dedupping and caching is less useful now, and
> we are complicating the inner implementations for forwarding a lot. We need
> to re-consider this feature with finer granularity of turning on / off
> caching per store, potentially with explicit triggers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-4587) Rethink Unification of Caching with Dedupping
[
https://issues.apache.org/jira/browse/KAFKA-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069241#comment-17069241
]
Maatari commented on KAFKA-4587:
Hi all, i'm note sure to follow what is the status of this ticket. Is it still
relevant ?
> Rethink Unification of Caching with Dedupping
> -
>
> Key: KAFKA-4587
> URL: https://issues.apache.org/jira/browse/KAFKA-4587
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Priority: Major
>
> This is discussed in PR https://github.com/apache/kafka/pull/1588
> In order to support user-customizable state store suppliers in the DSL we did
> the following:
> 1) Introduce a {{TupleForwarder}} to forward tuples from cached stores that
> is wrapping user customized stores.
> 2) Narrow the scope to only dedup on forwarding if it is the default
> CachingXXStore with wrapper RocksDB.
> With this, the unification of dedupping and caching is less useful now, and
> we are complicating the inner implementations for forwarding a lot. We need
> to re-consider this feature with finer granularity of turning on / off
> caching per store, potentially with explicit triggers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-4587) Rethink Unification of Caching with Dedupping
[
https://issues.apache.org/jira/browse/KAFKA-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770587#comment-16770587
]
Matthias J. Sax commented on KAFKA-4587:
Moving all major/minor/trivial tickets that are not merged yet out of 2.2
release.
> Rethink Unification of Caching with Dedupping
> -
>
> Key: KAFKA-4587
> URL: https://issues.apache.org/jira/browse/KAFKA-4587
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Priority: Major
> Fix For: 2.2.0
>
>
> This is discussed in PR https://github.com/apache/kafka/pull/1588
> In order to support user-customizable state store suppliers in the DSL we did
> the following:
> 1) Introduce a {{TupleForwarder}} to forward tuples from cached stores that
> is wrapping user customized stores.
> 2) Narrow the scope to only dedup on forwarding if it is the default
> CachingXXStore with wrapper RocksDB.
> With this, the unification of dedupping and caching is less useful now, and
> we are complicating the inner implementations for forwarding a lot. We need
> to re-consider this feature with finer granularity of turning on / off
> caching per store, potentially with explicit triggers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-4587) Rethink Unification of Caching with Dedupping
[
https://issues.apache.org/jira/browse/KAFKA-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608655#comment-16608655
]
Matthias J. Sax commented on KAFKA-4587:
[~guozhang] I think we should move this to fixed version 2.2.0 ?
> Rethink Unification of Caching with Dedupping
> -
>
> Key: KAFKA-4587
> URL: https://issues.apache.org/jira/browse/KAFKA-4587
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Guozhang Wang
>Priority: Major
> Fix For: 2.1.0
>
>
> This is discussed in PR https://github.com/apache/kafka/pull/1588
> In order to support user-customizable state store suppliers in the DSL we did
> the following:
> 1) Introduce a {{TupleForwarder}} to forward tuples from cached stores that
> is wrapping user customized stores.
> 2) Narrow the scope to only dedup on forwarding if it is the default
> CachingXXStore with wrapper RocksDB.
> With this, the unification of dedupping and caching is less useful now, and
> we are complicating the inner implementations for forwarding a lot. We need
> to re-consider this feature with finer granularity of turning on / off
> caching per store, potentially with explicit triggers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
