----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27396/#review60601 -----------------------------------------------------------
Ship it! Ship It! - Nate Cole On Nov. 6, 2014, 9:11 p.m., Cabir Zounaidou wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27396/ > ----------------------------------------------------------- > > (Updated Nov. 6, 2014, 9:11 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John > Speidel, Mahadev Konar, Nate Cole, and Yusaku Sako. > > > Bugs: AMBARI-1522 > https://issues.apache.org/jira/browse/AMBARI-1522 > > > Repository: ambari > > > Description > ------- > > The idea is to run a scheduler which computes diffs in a component log and > pushes the diff (only if available) to HDFS at fixed interval. > > 1. Each component will have its own monitor > 2. The apscheduler threadpool controls the monitor execution. > 3. If the log directory is not available if the component (or service) is not > running, it will skip the processing. > 4. It saves the last read line index and last modified time for next iteration > 5. It uses HDFS shell utility to push log patch to HDFS. > 6. Right now, the component log directory is configured in a json file. In > the next iteration, it will try to automatically detected from the stack > config. > > > Diffs > ----- > > ambari-agent/src/main/python/ambari_agent/AmbariConfig.py ca2e80c > ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf > ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py > PRE-CREATION > ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION > ambari-agent/src/test/python/ambari_agent/TestHdfsApi.py PRE-CREATION > ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION > > ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json > PRE-CREATION > > Diff: https://reviews.apache.org/r/27396/diff/ > > > Testing > ------- > > - Ran the ambari-agent with the patch and verified > 1. Started ambari-agent with the patch > 2. The scheduler started successfully > 3. The logs files are getting pushed to HDFS successfully. Verified using > HDFS shell utility. > - Ran the tests successfully > [INFO] > [INFO] > ------------------------------------------------------------------------ > [INFO] Building Ambari Agent 1.3.0-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > [INFO] > : > ---------------------------------------------------------------------- > Ran 324 tests in 8.282s > > OK > : > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 14.759s > [INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014 > [INFO] Final Memory: 10M/4079M > [INFO] > ------------------------------------------------------------------------ > > > Thanks, > > Cabir Zounaidou > >
