[
https://issues.apache.org/jira/browse/SAMZA-144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Fang updated SAMZA-144:
---------------------------
Attachment: SAMZA-144.patch
RB: https://reviews.apache.org/r/22276/
1. added the stream-level override for fetch size.
2. basic idea is to use a tuple instead of a string for fetch size. First
element is the default value (or shared value if having system-level override),
second element is the map of topic->fetchsize. More explaination in RB.
3. tested the value is passed to DefaultFetchSimpleConsumer and the system
works. But did not have performance test showing the effectiveness of the
change.
Thank you.
> Add stream-level overrides for KafkaSystemFactory config
> --------------------------------------------------------
>
> Key: SAMZA-144
> URL: https://issues.apache.org/jira/browse/SAMZA-144
> Project: Samza
> Issue Type: Bug
> Components: kafka
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Assignee: Yan Fang
> Attachments: SAMZA-144.patch
>
>
> Currently, we allow Kafka systems to be defined with:
> {noformat}
> systems.my-kafka-system.consumer.*
> {noformat}
> And:
> {noformat}
> systems.my-kafka-system.producer.*
> {noformat}
> We don't support stream-level configurations, the way we do with StreamConfig
> configuration:
> {code}
> val STREAM_PREFIX = "systems.%s.streams.%s."
> val MSG_SERDE = STREAM_PREFIX + "samza.msg.serde"
> val KEY_SERDE = STREAM_PREFIX + "samza.key.serde"
> val CONSUMER_RESET_OFFSET = STREAM_PREFIX + "samza.reset.offset"
> {code}
> We should add stream-level configuration override support for Kafka consumer
> and producers with:
> {noformat}
> systems.my-kafka-system.streams.my-kafka-topic.consumer.*
> systems.my-kafka-system.streams.my-kafka-topic.producer.*
> {noformat}
> KafkaConfig and KafkaSystemFactory should be updated to handle these.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
