[
https://issues.apache.org/jira/browse/SAMZA-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167567#comment-14167567
]
Roger Hoover commented on SAMZA-40:
-----------------------------------
Thank you, Chris. That makes sense. For me, there seems to be an intermediate
between a job and a topology. If you talk about a topology that span multiple
teams like you mentioned, then it totally makes sense to have the loose
coupling. However, there's still a mini-topology that you need for a single
logical job. My logical jobs is to join a stream with a table but I actually
need 3 jobs to do this in Samza, two of those jobs will coherently partition
the "public" input topics into the right number of partitions that my stream
join job needs (with the same key -> partition id function).
Those 3 jobs would always need to be packaged and deployed together. Do you
guys see this as a common pattern?
> Refactor Samza configuration
> ----------------------------
>
> Key: SAMZA-40
> URL: https://issues.apache.org/jira/browse/SAMZA-40
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Labels: project
>
> Samza's configuration system has several problems that we need to resolved.
> * Want to auto-generate documentation based off of configuration.
> * Should support global defaults for a config property. Right now, we do
> config.getFoo.getOrElse() everywhere.
> * Should validate config up front, rather than thrown runtime exceptions
> randomly throughout the code.
> * We are mixing wiring and configuration together. How do other systems
> handle this?
> * We have fragmented configuration (anybody can define configuration). How do
> other systems handle this?
> * How to handle undefined configuration? How to make this interoperable with
> both Java and Scala (i.e. should we support Option in Scala)?
> * Should remain immutable.
> * Should remove implicits. It's just confusing.
> * Do we want to support complex types (list, map) for values, not just String?
> We need a design proposal for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
