----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58424/#review175038 -----------------------------------------------------------
Without goind throught full patch it is always a good idea to have a writeLock for emptying the cache, +1 Aravindan's idea of simply cachce and empty and drop on the floor if cannot submit. - Sid Wagle On April 28, 2017, 4:41 p.m., Dmytro Sen wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58424/ > ----------------------------------------------------------- > > (Updated April 28, 2017, 4:41 p.m.) > > > Review request for Ambari, Aravindan Vijayan and Sid Wagle. > > > Bugs: AMBARI-20758 > https://issues.apache.org/jira/browse/AMBARI-20758 > > > Repository: ambari > > > Description > ------- > > Abstract Sink checks if there is local known monitor port available to write > (e.g.: 61888) > Handshake / Ping to make sure its the monitor and is accepting writes > Start sending metrics only to the monitor > Monitor sends metrics at the global send interval (1 minute) with all the > granularity of the sink > Monitor aggregates first interval (5 min) data and sends it every interval > seconds. (5 min) > The monitor interface look the same as that of collector (http server: maybe > Flask) > Security concerns need to be addressed as well. (https/spnego?) > > > Diffs > ----- > > > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java > 2d1bf40 > ambari-metrics/ambari-metrics-assembly/pom.xml a4b87de > > ambari-metrics/ambari-metrics-assembly/src/main/assembly/monitor-windows.xml > ab309a1 > ambari-metrics/ambari-metrics-assembly/src/main/assembly/monitor.xml > 99a41c3 > > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/AbstractTimelineMetricsSink.java > 2c6fae2 > > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/AggregationResult.java > PRE-CREATION > > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/MetricAggregate.java > PRE-CREATION > > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/MetricClusterAggregate.java > PRE-CREATION > > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/MetricHostAggregate.java > PRE-CREATION > > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/TimelineMetric.java > 44c9d4a > > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/TimelineMetricWithAggregatedValues.java > PRE-CREATION > > ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/availability/AbstractTimelineMetricSinkTest.java > 9b0cdbe > > ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/availability/MetricCollectorHATest.java > a393a96 > > ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/HandleConnectExceptionTest.java > 32fe32e > > ambari-metrics/ambari-metrics-flume-sink/src/main/java/org/apache/hadoop/metrics2/sink/flume/FlumeTimelineMetricsSink.java > 904c916 > > ambari-metrics/ambari-metrics-hadoop-sink/src/main/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSink.java > 11e16c2 > ambari-metrics/ambari-metrics-host-aggregator/conf/unix/log4j.properties > PRE-CREATION > ambari-metrics/ambari-metrics-host-aggregator/conf/windows/log4j.properties > PRE-CREATION > ambari-metrics/ambari-metrics-host-aggregator/pom.xml PRE-CREATION > > ambari-metrics/ambari-metrics-host-aggregator/src/main/java/org/apache/hadoop/metrics2/host/aggregator/AbstractMetricPublisherThread.java > PRE-CREATION > > ambari-metrics/ambari-metrics-host-aggregator/src/main/java/org/apache/hadoop/metrics2/host/aggregator/AggregatedMetricsPublisher.java > PRE-CREATION > > ambari-metrics/ambari-metrics-host-aggregator/src/main/java/org/apache/hadoop/metrics2/host/aggregator/AggregatorApplication.java > PRE-CREATION > > ambari-metrics/ambari-metrics-host-aggregator/src/main/java/org/apache/hadoop/metrics2/host/aggregator/AggregatorWebService.java > PRE-CREATION > > ambari-metrics/ambari-metrics-host-aggregator/src/main/java/org/apache/hadoop/metrics2/host/aggregator/RawMetricsPublisher.java > PRE-CREATION > > ambari-metrics/ambari-metrics-host-aggregator/src/main/java/org/apache/hadoop/metrics2/host/aggregator/TimelineMetricsHolder.java > PRE-CREATION > > ambari-metrics/ambari-metrics-host-monitoring/conf/unix/ambari-metrics-monitor > 967e133 > > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/aggregator.py > PRE-CREATION > > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py > 2670e76 > > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/controller.py > c0feed5 > > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/emitter.py > e2a7f0d > > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/stop_handler.py > bfb6957 > ambari-metrics/ambari-metrics-host-monitoring/src/main/python/main.py > d218015 > > ambari-metrics/ambari-metrics-kafka-sink/src/main/java/org/apache/hadoop/metrics2/sink/kafka/KafkaTimelineMetricsReporter.java > 211e9cd > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > 08f0598 > > ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > 20f60e1 > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java > 14f160b > > ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java > 425201c > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java > c242a2f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java > fb369e8 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java > 0d5042f > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java > bde09cb > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorSink.java > 65d54c0 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/MetricAggregate.java > 825ac25 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/MetricClusterAggregate.java > 9c837b6 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/MetricHostAggregate.java > 340ec75 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricAppAggregator.java > 44aca03 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregator.java > 02677b9 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecond.java > a5a3499 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricHostAggregator.java > 0ea9c08 > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java > b5f49fb > > ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TimelineWebServices.java > 9da921a > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITPhoenixHBaseAccessor.java > 0087fd9 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/MetricTestHelper.java > 37ec134 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java > a910cc2 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestMetricHostAggregate.java > 44f48e8 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestTimelineMetricStore.java > f00906e > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsAggregatorMemorySink.java > fa0cfe9 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/ITClusterAggregator.java > 590f82a > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/ITMetricAggregator.java > 9873643 > > ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecondTest.java > 78db11d > ambari-metrics/pom.xml 2d88912 > > ambari-server/src/main/java/org/apache/ambari/server/metrics/system/impl/AmbariMetricSinkImpl.java > 8d1f63f > > ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/params.py > 150b0a8 > > ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/templates/hadoop-metrics2-accumulo.properties.j2 > 6873c85 > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml > cb66537 > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml > 8e1671e > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/metainfo.xml > 740a91a > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py > a929847 > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py > 50dde1c > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/hadoop-metrics2-hbase.properties.j2 > 9729bbe > > ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2 > 769ad67 > > ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/params.py > 86a290f > > ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/templates/flume-metrics2.properties.j2 > b876a3d > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py > efea167 > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2 > 24535c5 > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2 > 9076269 > > ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/configuration/hadoop-metrics2.properties.xml > fae61d3 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py > 0f03155 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/templates/hadoop-metrics2-hivemetastore.properties.j2 > 9328f9f > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/templates/hadoop-metrics2-hiveserver2.properties.j2 > 9a7f9dc > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/templates/hadoop-metrics2-llapdaemon.j2 > e9fe024 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/templates/hadoop-metrics2-llaptaskscheduler.j2 > bd7eb0c > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/params_linux.py > 69ef08e > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/templates/hadoop-metrics2-hivemetastore.properties.j2 > 9328f9f > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/templates/hadoop-metrics2-hiveserver2.properties.j2 > 9a7f9dc > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/templates/hadoop-metrics2-llapdaemon.j2 > e9fe024 > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/templates/hadoop-metrics2-llaptaskscheduler.j2 > bd7eb0c > > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/configuration/kafka-broker.xml > e01dacd > > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py > 5b0be54 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py > d9fae76 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/templates/config.yaml.j2 > 51162e8 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/templates/storm-metrics2.properties.j2 > 0501039 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py > 7282bb5 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2 > 1b02a97 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/hadoop-metrics2.properties.xml > fae61d3 > > ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-START/scripts/params.py > 678bbde > > ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-START/templates/hadoop-metrics2.properties.j2 > 1b02a97 > > ambari-server/src/test/java/org/apache/ambari/server/metric/system/impl/TestAmbariMetricsSinkImpl.java > 4a60892 > > contrib/management-packs/hdf-ambari-mpack/src/main/resources/stacks/HDF/2.0/hooks/before-START/scripts/params.py > 6a70675 > > > Diff: https://reviews.apache.org/r/58424/diff/4/ > > > Testing > ------- > > Under development > > > Thanks, > > Dmytro Sen > >
