----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27396/ -----------------------------------------------------------
(Updated Oct. 31, 2014, 11:50 p.m.) Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John Speidel, Mahadev Konar, Nate Cole, and Yusaku Sako. Changes ------- Added update_config method to reschedule jobs. Bugs: AMBARI-1522 https://issues.apache.org/jira/browse/AMBARI-1522 Repository: ambari Description ------- The idea is to run a scheduler which computes diffs in a component log and pushes the diff (only if available) to HDFS at fixed interval. 1. Each component will have its own monitor 2. The apscheduler threadpool controls the monitor execution. 3. If the log directory is not available if the component (or service) is not running, it will skip the processing. 4. It saves the last read line index and last modified time for next iteration 5. It uses HDFS shell utility to push log patch to HDFS. 6. Right now, the component log directory is configured in a json file. In the next iteration, it will try to automatically detected from the stack config. Diffs (updated) ----- ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py PRE-CREATION ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py PRE-CREATION ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py PRE-CREATION ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py PRE-CREATION ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json PRE-CREATION Diff: https://reviews.apache.org/r/27396/diff/ Testing ------- - Ran the ambari-agent with the patch and verified 1. Started ambari-agent with the patch 2. The scheduler started successfully 3. The logs files are getting pushed to HDFS successfully. Verified using HDFS shell utility. - Ran the tests successfully [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Ambari Agent 1.3.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] : ---------------------------------------------------------------------- Ran 324 tests in 8.282s OK : [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 14.759s [INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014 [INFO] Final Memory: 10M/4079M [INFO] ------------------------------------------------------------------------ Thanks, Cabir Zounaidou