[ https://issues.apache.org/jira/browse/SAMZA-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220302#comment-15220302 ]
Yi Pan (Data Infrastructure) commented on SAMZA-348: ---------------------------------------------------- Hi, [~alex.buck10], I have created the sub-task SAMZA-921 and assigned to you. There have been many updates regarding to the CoordinatorStream and the JobCoordinator recently. Please check SAMZA-448 for the implementation of CoordinatorStream. This might also be related to the refactoring of JobCoordinator that we are actively working on: SAMZA-881. So, it would be good if the design/implementation of the dynamic re-config via restart can be compatible w/ the refactored JobCoordinator as well. As for the bugs you found in JSON deserialization, feel free to open a JIRA. Everyone should have the power to open JIRA in Samza. Thanks a lot! > Configure Samza jobs through a stream > ------------------------------------- > > Key: SAMZA-348 > URL: https://issues.apache.org/jira/browse/SAMZA-348 > Project: Samza > Issue Type: Bug > Affects Versions: 0.7.0 > Reporter: Chris Riccomini > Assignee: Chris Riccomini > Labels: design, project > Attachments: DESIGN-SAMZA-348-0.md, DESIGN-SAMZA-348-0.pdf, > DESIGN-SAMZA-348-1.md, DESIGN-SAMZA-348-1.pdf > > > Samza's existing config setup is problematic for a number of reasons: > # It's completely immutable once a job starts. This prevents any dynamic > reconfiguration and auto-scaling. It is debatable whether we want these > feature or not, but our existing implementation actively prevents it. See > SAMZA-334 for discussion. > # We pass existing configuration through environment variables. YARN exports > environment variables in a shell script, which limits the size to the varargs > length on the machine. This is usually ~128KB. See SAMZA-333 and SAMZA-337 > for details. > # User-defined configuration (the Config object) and programmatic > configuration (checkpoints and TaskName:State mappings (see SAMZA-123)) are > handled differently. It's debatable whether this makes sense. > In SAMZA-123, [~jghoman] and I propose implementing a ConfigLog. This log > would replace both the checkpoint topic and the existing config environment > variables in SamzaContainer and Samza's YARN AM. > I'd like to keep this ticket's scope limited to just the implementation of > the ConfigLog, and not re-designing how Samza's config is used in the code > (SAMZA-40). We should, however, discuss how this feature would affect dynamic > reconfiguration/auto-scaling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)