[jira] [Commented] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16355257#comment-16355257
]
james chien commented on KAFKA-6498:
[~guozhang] Now I familiar with kafka metrics, so I want to confirm the goals,
the following two points are my thinking about to be resolved.
1) For currency, we enumerate all of metrics but not to let user select
specific metrics directly, so we have to add this.
2) Design a new interface to allow user manually add more `rocksDB metrics` as
you mentioned above in a custom way.
If both of two are right, I will be making a KIP proposing.
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Assignee: james chien
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351245#comment-16351245
]
james chien commented on KAFKA-6498:
[~guozhang] Thank for your summarization, I will start to do the research and
come up with a better way to accomplish and write a KIP proposing also.
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Assignee: james chien
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347963#comment-16347963
]
james chien edited comment on KAFKA-6498 at 2/2/18 9:01 AM:
[~guozhang] I want to pick it up!
For my comprehension is that we want to expose RockDB stats via Streams metrics
so that user can use Streams metrics simply to monitor RockDB stats but not to
access RockDB directly.
was (Author: james.c):
[~guozhang] I want to pick it up!
For my comprehension is that we want to expose RockDB stats via Streams metrics
so that user can use Streams metrics simply to monitor RockDB stats but not to
access RockDB directly which means `KStream` should be extended.
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Assignee: james chien
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347963#comment-16347963
]
james chien edited comment on KAFKA-6498 at 2/2/18 8:47 AM:
[~guozhang] I want to pick it up!
For my comprehension is that we want to expose RockDB stats via Streams metrics
so that user can use Streams metrics simply to monitor RockDB stats but not to
access RockDB directly which means `KStream` should be extended.
was (Author: james.c):
[~guozhang] I want to pick it up!
As my comprehension is that we want to expose RockDB stats via Streams metrics
so that user can use Streams metrics simply to monitor RockDB stats but not to
access RockDB directly.
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Assignee: james chien
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
james chien updated KAFKA-6498:
---
Comment: was deleted
(was: I have one more question is that source of stats is limited in db of
kafka?)
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Assignee: james chien
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348347#comment-16348347
]
james chien edited comment on KAFKA-6498 at 2/1/18 10:26 AM:
-
I have one more question is that source of stats is limited in db of kafka?
was (Author: james.c):
I have one more question is that source of stats is limited in db of kafka or
any?
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Assignee: james chien
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348347#comment-16348347
]
james chien commented on KAFKA-6498:
I have one more question is that source of stats is limited in db of kafka or
any?
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Assignee: james chien
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
james chien reassigned KAFKA-6498:
--
Assignee: james chien
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Assignee: james chien
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347963#comment-16347963
]
james chien commented on KAFKA-6498:
[~guozhang] I want to pick it up!
As my comprehension is that we want to expose RockDB stats via Streams metrics
so that user can use Streams metrics simply to monitor RockDB stats but not to
access RockDB directly.
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6498) Add RocksDB statistics via Streams metrics
[
https://issues.apache.org/jira/browse/KAFKA-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347019#comment-16347019
]
james chien commented on KAFKA-6498:
I have interest to this, but need to figure out how to do this.
> Add RocksDB statistics via Streams metrics
> --
>
> Key: KAFKA-6498
> URL: https://issues.apache.org/jira/browse/KAFKA-6498
> Project: Kafka
> Issue Type: Improvement
> Components: metrics, streams
>Reporter: Guozhang Wang
>Priority: Major
> Labels: needs-kip
>
> RocksDB's own stats can be programmatically exposed via
> {{Options.statistics()}} and the JNI `Statistics` has indeed implemented many
> useful settings already. However these stats are not exposed directly via
> Streams today and hence for any users who wants to get access to them they
> have to manually interact with the underlying RocksDB directly, not through
> Streams.
> We should expose such stats via Streams metrics programmatically for users to
> investigate them without trying to access the rocksDB directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6048) Support negative record timestamps
[
https://issues.apache.org/jira/browse/KAFKA-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270062#comment-16270062
]
james chien commented on KAFKA-6048:
[~mjsax] ok!!
> Support negative record timestamps
> --
>
> Key: KAFKA-6048
> URL: https://issues.apache.org/jira/browse/KAFKA-6048
> Project: Kafka
> Issue Type: Improvement
> Components: clients, core, streams
>Affects Versions: 1.0.0
>Reporter: Matthias J. Sax
>Assignee: Konstantin Chukhlomin
> Labels: needs-kip
>
> Kafka does not support negative record timestamps, and this prevents the
> storage of historical data in Kafka. In general, negative timestamps are
> supported by UNIX system time stamps:
> From https://en.wikipedia.org/wiki/Unix_time
> {quote}
> The Unix time number is zero at the Unix epoch, and increases by exactly
> 86,400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after
> the epoch, is represented by the Unix time number 12,677 × 86,400 =
> 1095292800. This can be extended backwards from the epoch too, using negative
> numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch, is
> represented by the Unix time number −4,472 × 86,400 = −386380800.
> {quote}
> Allowing for negative timestamps would require multiple changes:
> - while brokers in general do support negative timestamps, broker use {{-1}}
> as default value if a producer uses an old message format (this would not be
> compatible with supporting negative timestamps "end-to-end" as {{-1}} cannot
> be used as "unknown" anymore): we could introduce a message flag indicating a
> missing timestamp (and let producer throw an exception if
> {{ConsumerRecord#timestamp()}} is called. Another possible solution might be,
> to require topics that are used by old producers to be configured with
> {{LogAppendTime}} semantics and rejecting writes to topics with
> {{CreateTime}} semantics for older message formats
> - {{KafkaProducer}} does not allow to send records with negative timestamp
> and thus this would need to be fixed
> - Streams API does drop records with negative timestamps (or fails by
> default) -- also, some internal store implementation for windowed stores
> assume that there are not negative timestamps to do range queries
> There might be other gaps we need to address. This is just a summary of issue
> coming to my mind atm.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6048) Support negative record timestamps
[
https://issues.apache.org/jira/browse/KAFKA-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201778#comment-16201778
]
james chien commented on KAFKA-6048:
ok, I will start to do this :)
> Support negative record timestamps
> --
>
> Key: KAFKA-6048
> URL: https://issues.apache.org/jira/browse/KAFKA-6048
> Project: Kafka
> Issue Type: Improvement
> Components: clients, core, streams
>Affects Versions: 1.0.0
>Reporter: Matthias J. Sax
>Assignee: james chien
> Labels: needs-kip
>
> Kafka does not support negative record timestamps, and this prevents the
> storage of historical data in Kafka. In general, negative timestamps are
> supported by UNIX system time stamps:
> From https://en.wikipedia.org/wiki/Unix_time
> {quote}
> The Unix time number is zero at the Unix epoch, and increases by exactly
> 86,400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after
> the epoch, is represented by the Unix time number 12,677 × 86,400 =
> 1095292800. This can be extended backwards from the epoch too, using negative
> numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch, is
> represented by the Unix time number −4,472 × 86,400 = −386380800.
> {quote}
> Allowing for negative timestamps would require multiple changes:
> - while brokers in general do support negative timestamps, broker use {{-1}}
> as default value if a producer uses an old message format (this would not be
> compatible with supporting negative timestamps "end-to-end" as {{-1}} cannot
> be used as "unknown" anymore): we could introduce a message flag indicating a
> missing timestamp (and let producer throw an exception if
> {{ConsumerRecord#timestamp()}} is called. Another possible solution might be,
> to require topics that are used by old producers to be configured with
> {{LogAppendTime}} semantics and rejecting writes to topics with
> {{CreateTime}} semantics for older message formats
> - {{KafkaProducer}} does not allow to send records with negative timestamp
> and thus this would need to be fixed
> - Streams API does drop records with negative timestamps (or fails by
> default) -- also, some internal store implementation for windowed stores
> assume that there are not negative timestamps to do range queries
> There might be other gaps we need to address. This is just a summary of issue
> coming to my mind atm.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (KAFKA-6048) Support negative record timestamps
[
https://issues.apache.org/jira/browse/KAFKA-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
james chien reassigned KAFKA-6048:
--
Assignee: james chien
> Support negative record timestamps
> --
>
> Key: KAFKA-6048
> URL: https://issues.apache.org/jira/browse/KAFKA-6048
> Project: Kafka
> Issue Type: Improvement
> Components: clients, core, streams
>Affects Versions: 1.0.0
>Reporter: Matthias J. Sax
>Assignee: james chien
> Labels: needs-kip
>
> Kafka does not support negative record timestamps, and this prevents the
> storage of historical data in Kafka. In general, negative timestamps are
> supported by UNIX system time stamps:
> From https://en.wikipedia.org/wiki/Unix_time
> {quote}
> The Unix time number is zero at the Unix epoch, and increases by exactly
> 86,400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after
> the epoch, is represented by the Unix time number 12,677 × 86,400 =
> 1095292800. This can be extended backwards from the epoch too, using negative
> numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch, is
> represented by the Unix time number −4,472 × 86,400 = −386380800.
> {quote}
> Allowing for negative timestamps would require multiple changes:
> - while brokers in general do support negative timestamps, broker use {{-1}}
> as default value if a producer uses an old message format (this would not be
> compatible with supporting negative timestamps "end-to-end" as {{-1}} cannot
> be used as "unknown" anymore): we could introduce a message flag indicating a
> missing timestamp (and let producer throw an exception if
> {{ConsumerRecord#timestamp()}} is called. Another possible solution might be,
> to require topics that are used by old producers to be configured with
> {{LogAppendTime}} semantics and rejecting writes to topics with
> {{CreateTime}} semantics for older message formats
> - {{KafkaProducer}} does not allow to send records with negative timestamp
> and thus this would need to be fixed
> - Streams API does drop records with negative timestamps (or fails by
> default) -- also, some internal store implementation for windowed stores
> assume that there are not negative timestamps to do range queries
> There might be other gaps we need to address. This is just a summary of issue
> coming to my mind atm.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6048) Support negative record timestamps
[
https://issues.apache.org/jira/browse/KAFKA-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199888#comment-16199888
]
james chien commented on KAFKA-6048:
I am interested in this. :D
Yes, It seems couples issue to address.
> Support negative record timestamps
> --
>
> Key: KAFKA-6048
> URL: https://issues.apache.org/jira/browse/KAFKA-6048
> Project: Kafka
> Issue Type: Improvement
> Components: clients, core, streams
>Affects Versions: 1.0.0
>Reporter: Matthias J. Sax
> Labels: needs-kip
>
> Kafka does not support negative record timestamps, and this prevents the
> storage of historical data in Kafka. In general, negative timestamps are
> supported by UNIX system time stamps:
> From https://en.wikipedia.org/wiki/Unix_time
> {quote}
> The Unix time number is zero at the Unix epoch, and increases by exactly
> 86,400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after
> the epoch, is represented by the Unix time number 12,677 × 86,400 =
> 1095292800. This can be extended backwards from the epoch too, using negative
> numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch, is
> represented by the Unix time number −4,472 × 86,400 = −386380800.
> {quote}
> Allowing for negative timestamps would require multiple changes:
> - while brokers in general do support negative timestamps, broker use {{-1}}
> as default value if a producer uses an old message format (this would not be
> compatible with supporting negative timestamps "end-to-end" as {{-1}} cannot
> be used as "unknown" anymore): we could introduce a message flag indicating a
> missing timestamp (and let producer throw an exception if
> {{ConsumerRecord#timestamp()}} is called. Another possible solution might be,
> to require topics that are used by old producers to be configured with
> {{LogAppendTime}} semantics and rejecting writes to topics with
> {{CreateTime}} semantics for older message formats
> - {{KafkaProducer}} does not allow to send records with negative timestamp
> and thus this would need to be fixed
> - Streams API does drop records with negative timestamps (or fails by
> default) -- also, some internal store implementation for windowed stores
> assume that there are not negative timestamps to do range queries
> There might be other gaps we need to address. This is just a summary of issue
> coming to my mind atm.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5606) Review consumer's RequestFuture usage pattern
[
https://issues.apache.org/jira/browse/KAFKA-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126750#comment-16126750
]
james chien commented on KAFKA-5606:
[~ijuma] I open pull request
here([https://github.com/apache/kafka/pull/3667/]), please review it :D
> Review consumer's RequestFuture usage pattern
> -
>
> Key: KAFKA-5606
> URL: https://issues.apache.org/jira/browse/KAFKA-5606
> Project: Kafka
> Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: james chien
> Fix For: 1.0.0
>
>
> KAFKA-5556 shows that we can perhaps tighten the usage pattern of the
> consumer's RequestFuture to avoid similar bugs in the future.
> Jason suggested:
> {quote}
> Another way to see this bug is a failure to ensure completion of the future.
> Had we done so, then we could have skipped the failed check. This is why it
> worked prior to the patch which added the timeout. The pattern should really
> be something like this:
> {code}
> if (future.isDone()) {
> if (future.succeeded()) {
> // handle success
> } else {
> // handle failure
> }
> }
> {code}
> I guess one benefit of the enum approach is that it forces you to ensure
> completion prior to checking any of the possible results. That said, I'm a
> bit more inclined to remove the isRetriable method and leave it to the caller
> to determine what is and is not retriable. Then the request future only has
> two completion states.
> {quote}
> An alternative is replacing succeeded and failed with a status method
> returning an enum with 3 states: SUCCEEDED, FAILED, RETRY (the enum approach
> mentioned above). This would make sense if we often have to handle these 3
> states differently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5684) KStreamPrintProcessor as customized KStreamPeekProcessor
[
https://issues.apache.org/jira/browse/KAFKA-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110481#comment-16110481
]
james chien commented on KAFKA-5684:
[~guozhang] I agree with your point, we must to check K/V are byte[] if mapper
is not provided(that means we do not need to execute
{{KStreamPrint#maybeDeserialize}} .
> KStreamPrintProcessor as customized KStreamPeekProcessor
>
>
> Key: KAFKA-5684
> URL: https://issues.apache.org/jira/browse/KAFKA-5684
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>Priority: Minor
>
> Hi,
> the {{KStreamPrintProcessor}} is implemented from scratch (from the
> {{AbstractProcessor}}) and the same for the related supplier.
> It looks to me that it's just a special {{KStreamPeekProcessor}} with
> forwardDownStream to false and that allows the possibility to specify Serdes
> instances used if key/values are bytes.
> At same time used by a {{print()}} method it provides a fast way to print
> data flowing through the pipeline (while using just {{peek()}} you need to
> write the code).
> I think that it could be useful to refactoring the {{KStreamPrintProcessor}}
> as derived from the {{KStreamPeekProcessor}} customizing its behavior.
> Thanks,
> Paolo.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5606) Review consumer's RequestFuture usage pattern
[
https://issues.apache.org/jira/browse/KAFKA-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095721#comment-16095721
]
james chien edited comment on KAFKA-5606 at 7/21/17 3:35 AM:
-
So the goal is returning type of enum instead of boolean, specifically is
following statement:
{code:java}
RequestFuture.succeeded()
{code} with an enum of SUCCEEDED,
{code:java}
RequestFuture.failed()
{code} with an enum of FAILED,
and
{code:java}
RequestFuture.isRetriable()
{code} with an enum of RETRY
was (Author: james.c):
So the goal is replacing
{code:java}
RequestFuture.succeeded()
{code} with enum of SUCCEEDED,
{code:java}
RequestFuture.failed()
{code} with enum of FAILED,
and
{code:java}
RequestFuture.isRetriable()
{code} with enum of RETRY ?
> Review consumer's RequestFuture usage pattern
> -
>
> Key: KAFKA-5606
> URL: https://issues.apache.org/jira/browse/KAFKA-5606
> Project: Kafka
> Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: james chien
> Fix For: 0.11.1.0
>
>
> KAFKA-5556 shows that we can perhaps tighten the usage pattern of the
> consumer's RequestFuture to avoid similar bugs in the future.
> Jason suggested:
> {quote}
> Another way to see this bug is a failure to ensure completion of the future.
> Had we done so, then we could have skipped the failed check. This is why it
> worked prior to the patch which added the timeout. The pattern should really
> be something like this:
> {code}
> if (future.isDone()) {
> if (future.succeeded()) {
> // handle success
> } else {
> // handle failure
> }
> }
> {code}
> I guess one benefit of the enum approach is that it forces you to ensure
> completion prior to checking any of the possible results. That said, I'm a
> bit more inclined to remove the isRetriable method and leave it to the caller
> to determine what is and is not retriable. Then the request future only has
> two completion states.
> {quote}
> An alternative is replacing succeeded and failed with a status method
> returning an enum with 3 states: SUCCEEDED, FAILED, RETRY (the enum approach
> mentioned above). This would make sense if we often have to handle these 3
> states differently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (KAFKA-5606) Review consumer's RequestFuture usage pattern
[
https://issues.apache.org/jira/browse/KAFKA-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
james chien reassigned KAFKA-5606:
--
Assignee: james chien
> Review consumer's RequestFuture usage pattern
> -
>
> Key: KAFKA-5606
> URL: https://issues.apache.org/jira/browse/KAFKA-5606
> Project: Kafka
> Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: james chien
> Fix For: 0.11.1.0
>
>
> KAFKA-5556 shows that we can perhaps tighten the usage pattern of the
> consumer's RequestFuture to avoid similar bugs in the future.
> Jason suggested:
> {quote}
> Another way to see this bug is a failure to ensure completion of the future.
> Had we done so, then we could have skipped the failed check. This is why it
> worked prior to the patch which added the timeout. The pattern should really
> be something like this:
> {code}
> if (future.isDone()) {
> if (future.succeeded()) {
> // handle success
> } else {
> // handle failure
> }
> }
> {code}
> I guess one benefit of the enum approach is that it forces you to ensure
> completion prior to checking any of the possible results. That said, I'm a
> bit more inclined to remove the isRetriable method and leave it to the caller
> to determine what is and is not retriable. Then the request future only has
> two completion states.
> {quote}
> An alternative is replacing succeeded and failed with a status method
> returning an enum with 3 states: SUCCEEDED, FAILED, RETRY (the enum approach
> mentioned above). This would make sense if we often have to handle these 3
> states differently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5606) Review consumer's RequestFuture usage pattern
[
https://issues.apache.org/jira/browse/KAFKA-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095721#comment-16095721
]
james chien commented on KAFKA-5606:
So the goal is replacing
{code:java}
RequestFuture.succeeded()
{code} with enum of SUCCEEDED,
{code:java}
RequestFuture.failed()
{code} with enum of FAILED,
and
{code:java}
RequestFuture.isRetriable()
{code} with enum of RETRY ?
> Review consumer's RequestFuture usage pattern
> -
>
> Key: KAFKA-5606
> URL: https://issues.apache.org/jira/browse/KAFKA-5606
> Project: Kafka
> Issue Type: Bug
>Reporter: Ismael Juma
> Fix For: 0.11.1.0
>
>
> KAFKA-5556 shows that we can perhaps tighten the usage pattern of the
> consumer's RequestFuture to avoid similar bugs in the future.
> Jason suggested:
> {quote}
> Another way to see this bug is a failure to ensure completion of the future.
> Had we done so, then we could have skipped the failed check. This is why it
> worked prior to the patch which added the timeout. The pattern should really
> be something like this:
> {code}
> if (future.isDone()) {
> if (future.succeeded()) {
> // handle success
> } else {
> // handle failure
> }
> }
> {code}
> I guess one benefit of the enum approach is that it forces you to ensure
> completion prior to checking any of the possible results. That said, I'm a
> bit more inclined to remove the isRetriable method and leave it to the caller
> to determine what is and is not retriable. Then the request future only has
> two completion states.
> {quote}
> An alternative is replacing succeeded and failed with a status method
> returning an enum with 3 states: SUCCEEDED, FAILED, RETRY (the enum approach
> mentioned above). This would make sense if we often have to handle these 3
> states differently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5606) Review consumer's RequestFuture usage pattern
[
https://issues.apache.org/jira/browse/KAFKA-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095715#comment-16095715
]
james chien commented on KAFKA-5606:
I am interest with this :D
> Review consumer's RequestFuture usage pattern
> -
>
> Key: KAFKA-5606
> URL: https://issues.apache.org/jira/browse/KAFKA-5606
> Project: Kafka
> Issue Type: Bug
>Reporter: Ismael Juma
> Fix For: 0.11.1.0
>
>
> KAFKA-5556 shows that we can perhaps tighten the usage pattern of the
> consumer's RequestFuture to avoid similar bugs in the future.
> Jason suggested:
> {quote}
> Another way to see this bug is a failure to ensure completion of the future.
> Had we done so, then we could have skipped the failed check. This is why it
> worked prior to the patch which added the timeout. The pattern should really
> be something like this:
> {code}
> if (future.isDone()) {
> if (future.succeeded()) {
> // handle success
> } else {
> // handle failure
> }
> }
> {code}
> I guess one benefit of the enum approach is that it forces you to ensure
> completion prior to checking any of the possible results. That said, I'm a
> bit more inclined to remove the isRetriable method and leave it to the caller
> to determine what is and is not retriable. Then the request future only has
> two completion states.
> {quote}
> An alternative is replacing succeeded and failed with a status method
> returning an enum with 3 states: SUCCEEDED, FAILED, RETRY (the enum approach
> mentioned above). This would make sense if we often have to handle these 3
> states differently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
