----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27396/#review59598 -----------------------------------------------------------
ambari-agent/src/main/python/ambari_agent/loghandler/config.py <https://reviews.apache.org/r/27396/#comment100891> You should be using an AmbariConfig instance and create a section in that file to handle setting all these. AmbariConfig is the "external property file" for things like this. ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py <https://reviews.apache.org/r/27396/#comment100896> We may have an HdfsExecute common library for things like this ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py <https://reviews.apache.org/r/27396/#comment100889> This style is generally deprecated for the string.format() method. ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py <https://reviews.apache.org/r/27396/#comment100899> Use the "with" syntax. ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py <https://reviews.apache.org/r/27396/#comment100890> No error handling here can leave an open file. Use this style and the file won't be left hanging. with open(self.__metadata_path, 'w') as file: file.write(...) ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py <https://reviews.apache.org/r/27396/#comment100893> See comment about AmbariConfig ambari-agent/src/main/python/ambari_agent/loghandler/util.py <https://reviews.apache.org/r/27396/#comment100894> All you have is statics. This doesn't need to be in a class, can just be loghandler/util with methods in it. - Nate Cole On Nov. 3, 2014, 3:30 p.m., Cabir Zounaidou wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27396/ > ----------------------------------------------------------- > > (Updated Nov. 3, 2014, 3:30 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John > Speidel, Mahadev Konar, Nate Cole, and Yusaku Sako. > > > Bugs: AMBARI-1522 > https://issues.apache.org/jira/browse/AMBARI-1522 > > > Repository: ambari > > > Description > ------- > > The idea is to run a scheduler which computes diffs in a component log and > pushes the diff (only if available) to HDFS at fixed interval. > > 1. Each component will have its own monitor > 2. The apscheduler threadpool controls the monitor execution. > 3. If the log directory is not available if the component (or service) is not > running, it will skip the processing. > 4. It saves the last read line index and last modified time for next iteration > 5. It uses HDFS shell utility to push log patch to HDFS. > 6. Right now, the component log directory is configured in a json file. In > the next iteration, it will try to automatically detected from the stack > config. > > > Diffs > ----- > > ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf > ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION > ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION > > ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json > PRE-CREATION > > Diff: https://reviews.apache.org/r/27396/diff/ > > > Testing > ------- > > - Ran the ambari-agent with the patch and verified > 1. Started ambari-agent with the patch > 2. The scheduler started successfully > 3. The logs files are getting pushed to HDFS successfully. Verified using > HDFS shell utility. > - Ran the tests successfully > [INFO] > [INFO] > ------------------------------------------------------------------------ > [INFO] Building Ambari Agent 1.3.0-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > [INFO] > : > ---------------------------------------------------------------------- > Ran 324 tests in 8.282s > > OK > : > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 14.759s > [INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014 > [INFO] Final Memory: 10M/4079M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Cabir Zounaidou > >