[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control
[
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468825#comment-15468825
]
Guozhang Wang commented on KAFKA-3478:
--
[~mjsax] Since separate JIRAs have been created for these two separate goals,
could we close this ticket then, or do you think there are still some
additional feature requests that are covered only in this ticket?
> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Guozhang Wang
> Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer
> control of the input streams, for example, with two input streams, one of
> them always reading from offset 0 upon (re)-starting, and the other reading
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer
> control over these frameworks.
> =
> A finer flow control could also be used to allow for populating a {{KTable}}
> (with an "initial" state) before starting the actual processing (this feature
> was ask for in the mailing list multiple times already). Even if it is quite
> hard to define, *when* the initial populating phase should end, this might
> still be useful. There would be the following possibilities:
> 1) an initial fixed time period for populating
>(it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> 4) a throughput threshold, ie, if the populating frequency falls below
> the threshold, the KTable is considered "finished"
> 5) maybe something else ??
> The API might look something like this
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without
> reading any other topics until see one record with timestamp 1000.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control
[
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458985#comment-15458985
]
Bill Bejeck commented on KAFKA-3478:
Makes sense, but I didn't realize that was the case, thanks.
> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Guozhang Wang
> Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer
> control of the input streams, for example, with two input streams, one of
> them always reading from offset 0 upon (re)-starting, and the other reading
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer
> control over these frameworks.
> =
> A finer flow control could also be used to allow for populating a {{KTable}}
> (with an "initial" state) before starting the actual processing (this feature
> was ask for in the mailing list multiple times already). Even if it is quite
> hard to define, *when* the initial populating phase should end, this might
> still be useful. There would be the following possibilities:
> 1) an initial fixed time period for populating
>(it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> 4) a throughput threshold, ie, if the populating frequency falls below
> the threshold, the KTable is considered "finished"
> 5) maybe something else ??
> The API might look something like this
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without
> reading any other topics until see one record with timestamp 1000.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control
[
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458831#comment-15458831
]
Matthias J. Sax commented on KAFKA-3478:
All newly creates "Streams" Jiras are assigned to him by default on creation.
This does not mean anything. Just reassign to yourself if you want to pick onw
up.
> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Guozhang Wang
> Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer
> control of the input streams, for example, with two input streams, one of
> them always reading from offset 0 upon (re)-starting, and the other reading
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer
> control over these frameworks.
> =
> A finer flow control could also be used to allow for populating a {{KTable}}
> (with an "initial" state) before starting the actual processing (this feature
> was ask for in the mailing list multiple times already). Even if it is quite
> hard to define, *when* the initial populating phase should end, this might
> still be useful. There would be the following possibilities:
> 1) an initial fixed time period for populating
>(it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> 4) a throughput threshold, ie, if the populating frequency falls below
> the threshold, the KTable is considered "finished"
> 5) maybe something else ??
> The API might look something like this
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without
> reading any other topics until see one record with timestamp 1000.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control
[
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458306#comment-15458306
]
Bill Bejeck commented on KAFKA-3478:
[~mjsax] thanks for the heads up, but it looks like they are assigned to
[~guozhang]
> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Guozhang Wang
> Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer
> control of the input streams, for example, with two input streams, one of
> them always reading from offset 0 upon (re)-starting, and the other reading
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer
> control over these frameworks.
> =
> A finer flow control could also be used to allow for populating a {{KTable}}
> (with an "initial" state) before starting the actual processing (this feature
> was ask for in the mailing list multiple times already). Even if it is quite
> hard to define, *when* the initial populating phase should end, this might
> still be useful. There would be the following possibilities:
> 1) an initial fixed time period for populating
>(it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> 4) a throughput threshold, ie, if the populating frequency falls below
> the threshold, the KTable is considered "finished"
> 5) maybe something else ??
> The API might look something like this
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without
> reading any other topics until see one record with timestamp 1000.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control
[
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456311#comment-15456311
]
Matthias J. Sax commented on KAFKA-3478:
[~bbejeck] I just created https://issues.apache.org/jira/browse/KAFKA-4113 and
https://issues.apache.org/jira/browse/KAFKA-4114. Feel free to grab one :)
> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Guozhang Wang
> Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer
> control of the input streams, for example, with two input streams, one of
> them always reading from offset 0 upon (re)-starting, and the other reading
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer
> control over these frameworks.
> =
> A finer flow control could also be used to allow for populating a {{KTable}}
> (with an "initial" state) before starting the actual processing (this feature
> was ask for in the mailing list multiple times already). Even if it is quite
> hard to define, *when* the initial populating phase should end, this might
> still be useful. There would be the following possibilities:
> 1) an initial fixed time period for populating
>(it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> 4) a throughput threshold, ie, if the populating frequency falls below
> the threshold, the KTable is considered "finished"
> 5) maybe something else ??
> The API might look something like this
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without
> reading any other topics until see one record with timestamp 1000.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control
[
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456310#comment-15456310
]
Matthias J. Sax commented on KAFKA-3478:
[~bbejeck] I just created https://issues.apache.org/jira/browse/KAFKA-4113 and
https://issues.apache.org/jira/browse/KAFKA-4114. Feel free to grab one :)
> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Guozhang Wang
> Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer
> control of the input streams, for example, with two input streams, one of
> them always reading from offset 0 upon (re)-starting, and the other reading
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer
> control over these frameworks.
> =
> A finer flow control could also be used to allow for populating a {{KTable}}
> (with an "initial" state) before starting the actual processing (this feature
> was ask for in the mailing list multiple times already). Even if it is quite
> hard to define, *when* the initial populating phase should end, this might
> still be useful. There would be the following possibilities:
> 1) an initial fixed time period for populating
>(it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> 4) a throughput threshold, ie, if the populating frequency falls below
> the threshold, the KTable is considered "finished"
> 5) maybe something else ??
> The API might look something like this
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without
> reading any other topics until see one record with timestamp 1000.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control
[
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429559#comment-15429559
]
Bill Bejeck commented on KAFKA-3478:
[~mjsax], thanks for the clarification. I'll take a look at what it would take
for different configurations, but I'll hold off doing anything concrete until
some of the details for this task (or subtasks) are fleshed out.
> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Guozhang Wang
> Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer
> control of the input streams, for example, with two input streams, one of
> them always reading from offset 0 upon (re)-starting, and the other reading
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer
> control over these frameworks.
> =
> A finer flow control could also be used to allow for populating a {{KTable}}
> (with an "initial" state) before starting the actual processing (this feature
> was ask for in the mailing list multiple times already). Even if it is quite
> hard to define, *when* the initial populating phase should end, this might
> still be useful. There would be the following possibilities:
> 1) an initial fixed time period for populating
>(it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> 4) a throughput threshold, ie, if the populating frequency falls below
> the threshold, the KTable is considered "finished"
> 5) maybe something else ??
> The API might look something like this
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without
> reading any other topics until see one record with timestamp 1000.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control
[
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429400#comment-15429400
]
Matthias J. Sax commented on KAFKA-3478:
Hey [~bbejeck]. It is still available. However, it is not clearly defined what
should be done. The description is more or less a collection of ideas rather
than defined things to do. We first need to make some design decisions and also
might want do define sub-tasks for individual things, too. Once thing to start,
would be to allow different configurations for different sources
IMHO.[~guozhang] [~miguno] [~enothereska] [~damianguy] [~hjafarpour] ?
> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Guozhang Wang
> Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer
> control of the input streams, for example, with two input streams, one of
> them always reading from offset 0 upon (re)-starting, and the other reading
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer
> control over these frameworks.
> =
> A finer flow control could also be used to allow for populating a {{KTable}}
> (with an "initial" state) before starting the actual processing (this feature
> was ask for in the mailing list multiple times already). Even if it is quite
> hard to define, *when* the initial populating phase should end, this might
> still be useful. There would be the following possibilities:
> 1) an initial fixed time period for populating
>(it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> 4) a throughput threshold, ie, if the populating frequency falls below
> the threshold, the KTable is considered "finished"
> 5) maybe something else ??
> The API might look something like this
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without
> reading any other topics until see one record with timestamp 1000.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control
[
https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422907#comment-15422907
]
Bill Bejeck commented on KAFKA-3478:
Is this task still available and is it a feature that is still in the current
plan/desired to get done?
> Finer Stream Flow Control
> -
>
> Key: KAFKA-3478
> URL: https://issues.apache.org/jira/browse/KAFKA-3478
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
>Reporter: Guozhang Wang
> Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to
> synchronize multiple input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer
> control of the input streams, for example, with two input streams, one of
> them always reading from offset 0 upon (re)-starting, and the other reading
> for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that
> behavior, which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer
> control over these frameworks.
> =
> A finer flow control could also be used to allow for populating a {{KTable}}
> (with an "initial" state) before starting the actual processing (this feature
> was ask for in the mailing list multiple times already). Even if it is quite
> hard to define, *when* the initial populating phase should end, this might
> still be useful. There would be the following possibilities:
> 1) an initial fixed time period for populating
>(it might be hard for a user to estimate the correct value)
> 2) an "idle" period, ie, if no update to a KTable for a certain time is
> done, we consider it as populated
> 3) a timestamp cut off point, ie, all records with an older timestamp
> belong to the initial populating phase
> 4) a throughput threshold, ie, if the populating frequency falls below
> the threshold, the KTable is considered "finished"
> 5) maybe something else ??
> The API might look something like this
> {noformat}
> KTable table = builder.table("topic", 1000); // populate the table without
> reading any other topics until see one record with timestamp 1000.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-3478) Finer Stream Flow Control
[ https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15306409#comment-15306409 ] Michael Noll commented on KAFKA-3478: - Latest documentation link (covers flow control with timestamps for Apache Kafka 0.10.0.0): http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps > Finer Stream Flow Control > - > > Key: KAFKA-3478 > URL: https://issues.apache.org/jira/browse/KAFKA-3478 > Project: Kafka > Issue Type: Sub-task > Components: streams >Reporter: Guozhang Wang > Labels: user-experience > Fix For: 0.10.1.0 > > > Today we have a event-time based flow control mechanism in order to > synchronize multiple input streams in a best effort manner: > http://docs.confluent.io/2.1.0-alpha1/streams/architecture.html#flow-control-with-timestamps > However, there are some use cases where users would like to have finer > control of the input streams, for example, with two input streams, one of > them always reading from offset 0 upon (re)-starting, and the other reading > for log end offset. > Today we only have one consumer config "offset.auto.reset" to control that > behavior, which means all streams are read either from "earliest" or "latest". > We should consider how to improve this settings to allow users have finer > control over these frameworks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
