[
https://issues.apache.org/jira/browse/SAMZA-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Riccomini updated SAMZA-2:
--------------------------------
Attachment: DESIGN-SAMZA-2-0.pdf
Migrating wiki-based SEP to .md/.pdf-based design doc as part of SAMZA-404.
> Fine-grain control over stream consumption
> ------------------------------------------
>
> Key: SAMZA-2
> URL: https://issues.apache.org/jira/browse/SAMZA-2
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Assignee: Chris Riccomini
> Fix For: 0.7.0
>
> Attachments: DESIGN-SAMZA-2-0.md, DESIGN-SAMZA-2-0.pdf,
> SAMZA-2.0.patch, SAMZA-2.1.patch, SAMZA-2.2.patch, SAMZA-2.3.patch,
> SAMZA-2.4.patch, SAMZA-2.5.patch
>
>
> Currently, samza exposes configuration in the form of
> "streams.%s.consumer.max.bytes.per.sec" for throttling the # of bytes the
> Task will read from a stream. This is a feature request for programmatic
> fine-grain control over stream consumption. The use-case is a samza task that
> will be consuming multiple streams where some streams may be from live
> systems that have stricter SLA requirements and must always be prioritized
> over other streams that may be from batch systems. The above configuration is
> not the ideal way to express this type of stream prioritization because
> configuring the "batch" streams with a low consumption rate will decrease the
> overall throughput of the system when there is no data in the "live" streams.
> Furthermore, we'll want to throttle each "batch" stream based on external
> signals that can change over time. Because of the dynamic nature of these
> external signals, we would like to have a programmatic interface that can
> dynamically change the prioritization as the signal changes.
> Design proposal:
> https://wiki.apache.org/samza/Pluggable%20MessageChooser
> Review board:
> https://reviews.apache.org/r/13725/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
