----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36006/#review92616 -----------------------------------------------------------
build.gradle (line 150) <https://reviews.apache.org/r/36006/#comment146824> We already are including jackson libraries as dependencies. What are additional functions that must require gson lib here? Generally, we want to be stingy in expanding out dependencies unless absolutely necessary. build.gradle (line 151) <https://reviews.apache.org/r/36006/#comment146825> Same question here. We want to make samza-core depends on less externals. I start to think that it might be reasonable to move the tools to a separate module like samza-man? samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala (line 52) <https://reviews.apache.org/r/36006/#comment146826> And how does the JobRunner knows whether it is starting the job the first time, if the first time is defined across job restarts? samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala (line 127) <https://reviews.apache.org/r/36006/#comment146828> Question: can we move this code into the JobCoordinator.start()? I don't see any reason why this has to be outside JobCoordinator? Some high-level comments: I don't see a reason that these tools classes have to be in samza-core. Can we move it to a separate module, like samza-man? That will also allow us to keep the minimum dependencies in samza-core. I will continue w/ the implementation of ConfigManager. - Yi Pan (Data Infrastructure) On July 18, 2015, 2:52 a.m., Shadi A. Noghabi wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/36006/ > ----------------------------------------------------------- > > (Updated July 18, 2015, 2:52 a.m.) > > > Review request for samza, Yi Pan (Data Infrastructure), Navina Ramesh, and > Naveen Somasundaram. > > > Repository: samza > > > Description > ------- > > After a job is submitted, it might need some configuration change, > specifically it might need more containers. In SAMZA-704 a tool is being > added to write to the coordinator stream (CoordinatorStreamWriter). This > tool can be used to write new configurations to the coordinator stream. > However, another tool (ConfigManager) is needed to read the config changes > and react to them, which is the goal of this task. This tool should be > brought up after the job is submitted and read any config changes added to > the coordinator stream, and react to each accordingly. > > This tool, called the Config Manager, is focusing on handling container > changs by reacting to set-config massages with key "yarn.container.count". > > The config manager is a separate standa alone module, that should be brought > up separately after the submission of a job. Therefore, you have to add two > configurations to the input config file: > 1. yarn.rm.address= <ip of resource manager in yarn. ex: localhost > > 2. yarn.rm.port= <the port of the resource manager http server. ex: 8088 > > > The config manger will periodically poll the coordinator stream to see if > there are any new messages. This period is set to 100 ms by deafualt. > However, it can be configured by adding > configManager.polling.interval=<polling interval> to the input config file. > Thus, overal the command to run the config manager along with the job would > be: > > > <path to samza deployment>/bin/run-config-manager.sh --config-factory=<config > factory> --config-path=<path to config file of a job> > > > Diffs > ----- > > build.gradle 0852adc4e8e0c2816afd1ebf433f1af6b44852f7 > checkstyle/import-control.xml 6654319392929857bb861d77763afd8a5ea7674c > gradle/dependency-versions.gradle fb06e8ed393d1a38abfa1a48fe5244fc7f6c7339 > > samza-core/src/main/java/org/apache/samza/autoScaling/deployer/ConfigManager.java > PRE-CREATION > > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java > b1078bdf7bddd16c9ccc6559b9efd40ca5ae67bc > samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala > 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 > samza-shell/src/main/bash/run-config-manager.sh PRE-CREATION > > samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala > ea702a919348305ff95ce0b4ca1996a13aff04ec > samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala > af42c6a6636953a95f79837fe372e0dbd735df70 > > samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMaster.scala > 7b7d86a43c69e72c47eaa91f68be24e0f4022891 > > Diff: https://reviews.apache.org/r/36006/diff/ > > > Testing > ------- > > > Thanks, > > Shadi A. Noghabi > >
