[jira] [Commented] (KAFKA-12766) Consider Disabling WAL-related Options in RocksDB
[ https://issues.apache.org/jira/browse/KAFKA-12766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403346#comment-17403346 ] Tomer Wizman commented on KAFKA-12766: -- [~cadonna] thanks! PR ready to review - https://github.com/apache/kafka/pull/11250 > Consider Disabling WAL-related Options in RocksDB > - > > Key: KAFKA-12766 > URL: https://issues.apache.org/jira/browse/KAFKA-12766 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Assignee: Tomer Wizman >Priority: Minor > Labels: newbie, newbie++ > Fix For: 3.1.0 > > > Streams disables the write-ahead log (WAL) provided by RocksDB since it > replicates the data in changelog topics. Hence, it does not make much sense > to set WAL-related configs for RocksDB instances within Streams. > Streams could: > - disable WAL-related options > - ignore WAL-related options > - throw an exception when a WAL-related option is set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12766) Consider Disabling WAL-related Options in RocksDB
[ https://issues.apache.org/jira/browse/KAFKA-12766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17402997#comment-17402997 ] Bruno Cadonna commented on KAFKA-12766: --- Hi [~TomerWizman]! Thank you for your interest! Since Streams disables the WAL and it will also log a warn message that the WAL is disabled and the WAL-related options are ignored, I would say that it shouldn't be too unexpected for users not getting the set WAL-related options. I would keep it simple and just ignore the set WAL-related options and state in the warn log message that they were ignored. Looking forward to your PR! > Consider Disabling WAL-related Options in RocksDB > - > > Key: KAFKA-12766 > URL: https://issues.apache.org/jira/browse/KAFKA-12766 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Assignee: Tomer Wizman >Priority: Minor > Labels: newbie, newbie++ > Fix For: 3.1.0 > > > Streams disables the write-ahead log (WAL) provided by RocksDB since it > replicates the data in changelog topics. Hence, it does not make much sense > to set WAL-related configs for RocksDB instances within Streams. > Streams could: > - disable WAL-related options > - ignore WAL-related options > - throw an exception when a WAL-related option is set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12766) Consider Disabling WAL-related Options in RocksDB
[ https://issues.apache.org/jira/browse/KAFKA-12766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17402624#comment-17402624 ] Tomer Wizman commented on KAFKA-12766: -- Hi Bruno, Sophie, Konstantine, nice e-meeting you :) According to the discussion here, the link Bruno attached and from what I saw in the code is that we explicitly disable WAL, so it doesn't really matter in terms of correctness whether we set the underlying Options object inside the adapter class or not. Choosing not to set it, getting null/empty value by calling the getters afterwards might seems a bit unexpected from user perspective. Let me know what you think (I will add warning messages to the log anyway) BTW this is my first task in Kafka Streams (and in open source projects in general), so any feedback is welcome! > Consider Disabling WAL-related Options in RocksDB > - > > Key: KAFKA-12766 > URL: https://issues.apache.org/jira/browse/KAFKA-12766 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Assignee: Tomer Wizman >Priority: Minor > Labels: newbie, newbie++ > Fix For: 3.1.0 > > > Streams disables the write-ahead log (WAL) provided by RocksDB since it > replicates the data in changelog topics. Hence, it does not make much sense > to set WAL-related configs for RocksDB instances within Streams. > Streams could: > - disable WAL-related options > - ignore WAL-related options > - throw an exception when a WAL-related option is set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12766) Consider Disabling WAL-related Options in RocksDB
[ https://issues.apache.org/jira/browse/KAFKA-12766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377751#comment-17377751 ] Konstantine Karantasis commented on KAFKA-12766: We are past feature freeze for 3.0 and this issue doesn't seem to be a bug. Postponing to the subsequent release. > Consider Disabling WAL-related Options in RocksDB > - > > Key: KAFKA-12766 > URL: https://issues.apache.org/jira/browse/KAFKA-12766 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Priority: Minor > Labels: newbie, newbie++ > Fix For: 3.0.0 > > > Streams disables the write-ahead log (WAL) provided by RocksDB since it > replicates the data in changelog topics. Hence, it does not make much sense > to set WAL-related configs for RocksDB instances within Streams. > Streams could: > - disable WAL-related options > - ignore WAL-related options > - throw an exception when a WAL-related option is set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12766) Consider Disabling WAL-related Options in RocksDB
[ https://issues.apache.org/jira/browse/KAFKA-12766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342337#comment-17342337 ] Bruno Cadonna commented on KAFKA-12766: --- I have to admit that I blindly followed your comment here: https://github.com/apache/kafka/pull/10568#discussion_r626058786 I interpreted "disable" as to not offer methods to read and write this option to users which seems hard to achieve. Now I see, that I misinterpreted. I am fine with ignoring. In this case, I would even log a warning if users try to read or write a WAL-related option since it is not obvious that Streams disables the WAL. > Consider Disabling WAL-related Options in RocksDB > - > > Key: KAFKA-12766 > URL: https://issues.apache.org/jira/browse/KAFKA-12766 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Priority: Minor > Fix For: 3.0.0 > > > Streams disables the write-ahead log (WAL) provided by RocksDB since it > replicates the data in changelog topics. Hence, it does not make much sense > to set WAL-related configs for RocksDB instances within Streams. > Streams could: > - disable WAL-related options > - ignore WAL-related options > - throw an exception when a WAL-related option is set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-12766) Consider Disabling WAL-related Options in RocksDB
[
https://issues.apache.org/jira/browse/KAFKA-12766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342123#comment-17342123
]
A. Sophie Blee-Goldman commented on KAFKA-12766:
Personally I agree that throwing an exception may be too harsh, it's not my
impression that any of the wal-related options are "critical" enough that a
user would want or need to be informed if they were going to be ignored. We
should definitely log a warning at the least.
Can you clarify a bit more on the difference between the first two approaches,
ie disable vs ignore? AFAICT they are pretty much the same -- since all user
options pass through our adapter class, if we just don't pass through to the
method on the underlying Option then we've both ignored and disabled it.
{quote}I do not understand how we should disable the option since we do not
have control over the `Options` object
{quote}
Based on this sentence I'm wondering if you had something else in mind by
"disable"? I feel that "ignoring" it may be sufficient, for the reasons
described above, plus it's simple to implement
> Consider Disabling WAL-related Options in RocksDB
> -
>
> Key: KAFKA-12766
> URL: https://issues.apache.org/jira/browse/KAFKA-12766
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Bruno Cadonna
>Priority: Minor
>
> Streams disables the write-ahead log (WAL) provided by RocksDB since it
> replicates the data in changelog topics. Hence, it does not make much sense
> to set WAL-related configs for RocksDB instances within Streams.
> Streams could:
> - disable WAL-related options
> - ignore WAL-related options
> - throw an exception when a WAL-related option is set.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-12766) Consider Disabling WAL-related Options in RocksDB
[ https://issues.apache.org/jira/browse/KAFKA-12766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17341782#comment-17341782 ] Bruno Cadonna commented on KAFKA-12766: --- I think throwing an exception is too harsh. Additionally, I do not understand how we should disable the option since we do not have control over the `Options` object. I think, if we want to do something about it, ignoring is the best solution. But I am also fine with not doing anything, because Streams disables the WAL anyways. > Consider Disabling WAL-related Options in RocksDB > - > > Key: KAFKA-12766 > URL: https://issues.apache.org/jira/browse/KAFKA-12766 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Bruno Cadonna >Priority: Minor > > Streams disables the write-ahead log (WAL) provided by RocksDB since it > replicates the data in changelog topics. Hence, it does not make much sense > to set WAL-related configs for RocksDB instances within Streams. > Streams could: > - disable WAL-related options > - ignore WAL-related options > - throw an exception when a WAL-related option is set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
