> On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote: > >
Thanks for the review. > On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote: > > ambari-agent/src/main/python/ambari_agent/loghandler/config.py, lines 27-49 > > <https://reviews.apache.org/r/27396/diff/2/?file=746521#file746521line27> > > > > You should be using an AmbariConfig instance and create a section in > > that file to handle setting all these. AmbariConfig is the "external > > property file" for things like this. Thanks for the suggestion. I will move this to AmbariConfig. > On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote: > > ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py, lines 71-75 > > <https://reviews.apache.org/r/27396/diff/2/?file=746522#file746522line71> > > > > We may have an HdfsExecute common library for things like this Will move to common library. > On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote: > > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py, lines 52-53 > > <https://reviews.apache.org/r/27396/diff/2/?file=746523#file746523line52> > > > > This style is generally deprecated for the string.format() method. Fixed locally. > On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote: > > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py, lines > > 166-173 > > <https://reviews.apache.org/r/27396/diff/2/?file=746523#file746523line166> > > > > Use the "with" syntax. Fixed locally. > On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote: > > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py, lines > > 178-180 > > <https://reviews.apache.org/r/27396/diff/2/?file=746523#file746523line178> > > > > No error handling here can leave an open file. Use this style and the > > file won't be left hanging. > > > > with open(self.__metadata_path, 'w') as file: > > file.write(...) Fixed locally. > On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote: > > ambari-agent/src/main/python/ambari_agent/loghandler/util.py, lines 30-33 > > <https://reviews.apache.org/r/27396/diff/2/?file=746524#file746524line30> > > > > All you have is statics. This doesn't need to be in a class, can just > > be loghandler/util with methods in it. Fixed. > On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote: > > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py, lines > > 183-189 > > <https://reviews.apache.org/r/27396/diff/2/?file=746523#file746523line183> > > > > See comment about AmbariConfig Will use AmbariConfig - Cabir ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27396/#review59598 ----------------------------------------------------------- On Nov. 4, 2014, 10:11 p.m., Cabir Zounaidou wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27396/ > ----------------------------------------------------------- > > (Updated Nov. 4, 2014, 10:11 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John > Speidel, Mahadev Konar, Nate Cole, and Yusaku Sako. > > > Bugs: AMBARI-1522 > https://issues.apache.org/jira/browse/AMBARI-1522 > > > Repository: ambari > > > Description > ------- > > The idea is to run a scheduler which computes diffs in a component log and > pushes the diff (only if available) to HDFS at fixed interval. > > 1. Each component will have its own monitor > 2. The apscheduler threadpool controls the monitor execution. > 3. If the log directory is not available if the component (or service) is not > running, it will skip the processing. > 4. It saves the last read line index and last modified time for next iteration > 5. It uses HDFS shell utility to push log patch to HDFS. > 6. Right now, the component log directory is configured in a json file. In > the next iteration, it will try to automatically detected from the stack > config. > > > Diffs > ----- > > ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf > ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION > ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION > > ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json > PRE-CREATION > > Diff: https://reviews.apache.org/r/27396/diff/ > > > Testing > ------- > > - Ran the ambari-agent with the patch and verified > 1. Started ambari-agent with the patch > 2. The scheduler started successfully > 3. The logs files are getting pushed to HDFS successfully. Verified using > HDFS shell utility. > - Ran the tests successfully > [INFO] > [INFO] > ------------------------------------------------------------------------ > [INFO] Building Ambari Agent 1.3.0-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > [INFO] > : > ---------------------------------------------------------------------- > Ran 324 tests in 8.282s > > OK > : > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 14.759s > [INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014 > [INFO] Final Memory: 10M/4079M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Cabir Zounaidou > >
