[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/709 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135794935 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml --- @@ -119,6 +119,45 @@ + METRON_PROFILER + Metron Profiler + MASTER + 1 + false + + + HBASE/HBASE_CLIENT + host + +true + + + + ZOOKEEPER/ZOOKEEPER_SERVER + cluster + +true + + + + KAFKA/KAFKA_BROKER + host + +true + + + + +scripts/profiler_master.py +PYTHON +600 + + +metron-profiler-env --- End diff -- Thanks! I'll add it now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135655011 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml --- @@ -119,6 +119,45 @@ + METRON_PROFILER + Metron Profiler + MASTER + 1 + false + + + HBASE/HBASE_CLIENT + host + +true + + + + ZOOKEEPER/ZOOKEEPER_SERVER + cluster + +true + + + + KAFKA/KAFKA_BROKER + host + +true + + + + +scripts/profiler_master.py +PYTHON +600 + + +metron-profiler-env --- End diff -- The comment from @nickwallen can be addressed by adding this line here: `metron-enrichment-env` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135279804 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py --- @@ -52,6 +52,26 @@ update_table = 'metron_update' update_cf = 't' +# Profiler +metron_profiler_topology = 'profiler' +profiler_input_topic = config['configurations']['metron-enrichment-env']['enrichment_output_topic'] +profiler_kafka_start = config['configurations']['metron-profiler-env']['profiler_kafka_start'] +profiler_period_duration = config['configurations']['metron-profiler-env']['profiler_period_duration'] +profiler_period_units = config['configurations']['metron-profiler-env']['profiler_period_units'] +profiler_ttl = config['configurations']['metron-profiler-env']['profiler_ttl'] --- End diff -- Yeah, I didn't quite entirely realize what the status.py was doing, so I went a bit overboard. I'll trim them back to just the stuff needed for status checks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135279546 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -251,6 +253,26 @@ threat_intel_join_parallelism = config['configurations']['metron-enrichment-env']['threat_intel_join_parallelism'] kafka_writer_parallelism = config['configurations']['metron-enrichment-env']['kafka_writer_parallelism'] +# Profiler + +metron_profiler_topology = 'profiler' +profiler_input_topic = config['configurations']['metron-enrichment-env']['enrichment_output_topic'] --- End diff -- It probably bears thinking about if we want to do the indexing topology this way too, honestly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/709#discussion_r135279244
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
---
@@ -214,6 +214,73 @@
}
},
{
+"name": "profiler",
--- End diff --
Yeah, I agree, we should have a consistent look and feel here. I'll
adjust. I think the REST panel looks like this and I cut and pasted. I'll
look into it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135264143 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -251,6 +253,26 @@ threat_intel_join_parallelism = config['configurations']['metron-enrichment-env']['threat_intel_join_parallelism'] kafka_writer_parallelism = config['configurations']['metron-enrichment-env']['kafka_writer_parallelism'] +# Profiler + +metron_profiler_topology = 'profiler' +profiler_input_topic = config['configurations']['metron-enrichment-env']['enrichment_output_topic'] --- End diff -- Its a good thing to bring up and discuss @merrimanr . I agree with @cestella . Doing it this way makes it less error-prone for the user. If the user changes the enrichment output topic, the Profiler will get auto-magically updated too. I cannot think of a use case where the Profiler does not want the output of Enrichment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135262154 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -202,6 +203,7 @@ client_jaas_path = metron_home + '/client_jaas.conf' client_jaas_arg = '-Djava.security.auth.login.config=' + metron_home + '/client_jaas.conf' enrichment_topology_worker_childopts = client_jaas_arg if security_enabled else '' +profiler_topology_worker_childopts = client_jaas_arg if security_enabled else '' --- End diff -- whoops, missed a step. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135261317 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py --- @@ -52,6 +52,26 @@ update_table = 'metron_update' update_cf = 't' +# Profiler +metron_profiler_topology = 'profiler' +profiler_input_topic = config['configurations']['metron-enrichment-env']['enrichment_output_topic'] +profiler_kafka_start = config['configurations']['metron-profiler-env']['profiler_kafka_start'] +profiler_period_duration = config['configurations']['metron-profiler-env']['profiler_period_duration'] +profiler_period_units = config['configurations']['metron-profiler-env']['profiler_period_units'] +profiler_ttl = config['configurations']['metron-profiler-env']['profiler_ttl'] --- End diff -- I found out the hard way, through experience. The ambari agent runs in a polling loop and calls the "status" function on the command script (profiler_master.py in this case). Not much documentation out there for this as far as I know. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135261379 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -251,6 +253,26 @@ threat_intel_join_parallelism = config['configurations']['metron-enrichment-env']['threat_intel_join_parallelism'] kafka_writer_parallelism = config['configurations']['metron-enrichment-env']['kafka_writer_parallelism'] +# Profiler + +metron_profiler_topology = 'profiler' +profiler_input_topic = config['configurations']['metron-enrichment-env']['enrichment_output_topic'] --- End diff -- So, the profiler's input topic, in metron, is the output of the enrichment topology. I did it this way to keep them in alignment. There is currently, to my mind, not a usecase where the profiler does take its data from the output of the enrichment topology. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135259877 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py --- @@ -52,6 +52,26 @@ update_table = 'metron_update' update_cf = 't' +# Profiler +metron_profiler_topology = 'profiler' +profiler_input_topic = config['configurations']['metron-enrichment-env']['enrichment_output_topic'] +profiler_kafka_start = config['configurations']['metron-profiler-env']['profiler_kafka_start'] +profiler_period_duration = config['configurations']['metron-profiler-env']['profiler_period_duration'] +profiler_period_units = config['configurations']['metron-profiler-env']['profiler_period_units'] +profiler_ttl = config['configurations']['metron-profiler-env']['profiler_ttl'] --- End diff -- Just a side question @merrimanr, how do you know that? Do you have documentation on what the status scripts are, when and who loads them? A guide? Asking for a friend. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135258100 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml --- @@ -0,0 +1,167 @@ + + + + + +profiler_kafka_start +UNCOMMITTED_EARLIEST +One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST +Input Topic Start + + value-list + + + EARLIEST + Earliest + + + LATEST + Latest + + + UNCOMMITTED_EARLIEST + Uncommitted Earliest + + + UNCOMMITTED_LATEST + Uncommitted Latest + + + 1 + + + +profiler_period_duration +15 +The duration of each profile period. This value should be defined along with profiler.period.duration.units +Period Duration + + +profiler_period_units +MINUTES +The units used to specify the profiler.period.duration. This value should be defined along with profiler.period.duration. +Period Units + + value-list + + + DAYS + Days + + + HOURS + Hours + + + MINUTES + Minutes + + + SECONDS + Seconds + + + 1 + + + + +profiler_ttl +30 +If a message has not been applied to a Profile in this period of time, the Profile will be terminated and its resources will be cleaned up. This value should be defined along with profiler.ttl.units. + This time-to-live does not affect the persisted Profile data in HBase. It only affects the state stored in memory during the execution of the latest profile period. This state will be deleted if the time-to-live is exceeded. + +Time to Live + + +profiler_ttl_units +MINUTES +The units used to specify the profiler.ttl. +Time To Live Units + + value-list + + + DAYS + Days + + + HOURS + Hours + + + MINUTES + Minutes + + + SECONDS + Seconds + + + 1 + + + + + +profiler_hbase_table +profiler +The name of the HBase table that profile data is written to. The Profiler expects that the table exists and is writable. +HBase Table + + +profiler_hbase_cf +P +The column family used to store profile data in HBase. +HBase Table Column Family + + +profiler_hbase_batch +10 +The number of puts that are written to HBase in a single batch. +HBase Batch Size + + +profiler_hbase_flush_interval +30 +The maximum number of seconds between batch writes to HBase. +HBase Flush Interval + + +profiler_topology_worker_childopts + +Extra topology child opts for the storm topology. +topology.worker.childopts + + true + + + +profiler_topology_workers +1 +The profiler storm topology workers +Number of Workers + + +profiler_topology_executors --- End diff -- This property is confusing to me. It ends up being bound to the Storm topology.acker.executors property in the profiler flux file. The indexing and enrichment topologies also have this property but it's named differently (indexing_acker_executors in the mpack and Enrichment Ackers in the Ambari UI for example). Whatever the correct naming convention is, I think they should be consistent. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135256865 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -251,6 +253,26 @@ threat_intel_join_parallelism = config['configurations']['metron-enrichment-env']['threat_intel_join_parallelism'] kafka_writer_parallelism = config['configurations']['metron-enrichment-env']['kafka_writer_parallelism'] +# Profiler + +metron_profiler_topology = 'profiler' +profiler_input_topic = config['configurations']['metron-enrichment-env']['enrichment_output_topic'] --- End diff -- Was this intentional? Wouldn't we want a configurable profiler input topic? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135256262 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py --- @@ -202,6 +203,7 @@ client_jaas_path = metron_home + '/client_jaas.conf' client_jaas_arg = '-Djava.security.auth.login.config=' + metron_home + '/client_jaas.conf' enrichment_topology_worker_childopts = client_jaas_arg if security_enabled else '' +profiler_topology_worker_childopts = client_jaas_arg if security_enabled else '' --- End diff -- As it stand, setting this property in Ambari won't have any effect. There needs to be an additional step that reads from the config and appends. See enrichment_topology_worker_childopts as an example. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r135255635 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py --- @@ -52,6 +52,26 @@ update_table = 'metron_update' update_cf = 't' +# Profiler +metron_profiler_topology = 'profiler' +profiler_input_topic = config['configurations']['metron-enrichment-env']['enrichment_output_topic'] +profiler_kafka_start = config['configurations']['metron-profiler-env']['profiler_kafka_start'] +profiler_period_duration = config['configurations']['metron-profiler-env']['profiler_period_duration'] +profiler_period_units = config['configurations']['metron-profiler-env']['profiler_period_units'] +profiler_ttl = config['configurations']['metron-profiler-env']['profiler_ttl'] --- End diff -- Do all of these need to be defined here? These are also defined in params_linux.py and most won't be used for status. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r134258314 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml --- @@ -0,0 +1,98 @@ + + + + + +profiler_kafka_start +UNCOMMITTED_EARLIEST +One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST +Input Topic Start + + +profiler_period_duration +15 +The duration of each profile period. This value should be defined along with profiler.period.duration.units +Period Duration + + +profiler_period_units +MINUTES +The units used to specify the profiler.period.duration. This value should be defined along with profiler.period.duration. +Period Units + + +profiler_ttl +30 +If a message has not been applied to a Profile in this period of time, the Profile will be terminated and its resources will be cleaned up. This value should be defined along with profiler.ttl.units. + This time-to-live does not affect the persisted Profile data in HBase. It only affects the state stored in memory during the execution of the latest profile period. This state will be deleted if the time-to-live is exceeded. + +Time to Live + + +profiler_ttl_units +MINUTES --- End diff -- +1, good catch. I'm a dork. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r134258288 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml --- @@ -0,0 +1,98 @@ + + + + + +profiler_kafka_start +UNCOMMITTED_EARLIEST +One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST +Input Topic Start + + +profiler_period_duration +15 +The duration of each profile period. This value should be defined along with profiler.period.duration.units +Period Duration + + --- End diff -- +1, good catch. I'm a dork. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r134258114 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json --- @@ -5,11 +5,13 @@ "_comment" : "dependencies for all cases", "METRON_INDEXING-INSTALL" : ["METRON_PARSERS-INSTALL"], "METRON_ENRICHMENT-INSTALL": ["METRON_INDEXING-INSTALL"], +"METRON_PROFILER-INSTALL": ["METRON_ENRICHMENT-INSTALL"], "METRON_REST-INSTALL": ["METRON_PARSERS-INSTALL"], "METRON_PARSERS-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_ENRICHMENT_MASTER-START"], "METRON_ENRICHMENT_MASTER-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START", "HBASE_MASTER-START", "HBASE_REGIONSERVER-START"], "METRON_ENRICHMENT_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_ENRICHMENT_MASTER-START"], "METRON_INDEXING-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_PARSERS-START"], +"METRON_PROFILER-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_INDEXING-START"], --- End diff -- Yeah, you're right. It's actually worse than what you suggest, it's overtly wrong: it needs zookeeper, storm, kafka, and hbase --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r134235349 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml --- @@ -0,0 +1,98 @@ + + + + + +profiler_kafka_start +UNCOMMITTED_EARLIEST +One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST +Input Topic Start + + +profiler_period_duration +15 +The duration of each profile period. This value should be defined along with profiler.period.duration.units +Period Duration + + +profiler_period_units +MINUTES +The units used to specify the profiler.period.duration. This value should be defined along with profiler.period.duration. +Period Units + + +profiler_ttl +30 +If a message has not been applied to a Profile in this period of time, the Profile will be terminated and its resources will be cleaned up. This value should be defined along with profiler.ttl.units. + This time-to-live does not affect the persisted Profile data in HBase. It only affects the state stored in memory during the execution of the latest profile period. This state will be deleted if the time-to-live is exceeded. + +Time to Live + + +profiler_ttl_units +MINUTES --- End diff -- Should constrain choice here too --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r134235333 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml --- @@ -0,0 +1,98 @@ + + + + + +profiler_kafka_start +UNCOMMITTED_EARLIEST +One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST +Input Topic Start + + +profiler_period_duration +15 +The duration of each profile period. This value should be defined along with profiler.period.duration.units +Period Duration + + --- End diff -- Should constrain choice here too --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #709: METRON-1122: Add support for the profiler in the m...
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/709#discussion_r134233234 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json --- @@ -5,11 +5,13 @@ "_comment" : "dependencies for all cases", "METRON_INDEXING-INSTALL" : ["METRON_PARSERS-INSTALL"], "METRON_ENRICHMENT-INSTALL": ["METRON_INDEXING-INSTALL"], +"METRON_PROFILER-INSTALL": ["METRON_ENRICHMENT-INSTALL"], "METRON_REST-INSTALL": ["METRON_PARSERS-INSTALL"], "METRON_PARSERS-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_ENRICHMENT_MASTER-START"], "METRON_ENRICHMENT_MASTER-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START", "HBASE_MASTER-START", "HBASE_REGIONSERVER-START"], "METRON_ENRICHMENT_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_ENRICHMENT_MASTER-START"], "METRON_INDEXING-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_PARSERS-START"], +"METRON_PROFILER-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_INDEXING-START"], --- End diff -- does it really need indexing running to start? (to work, yes, but to start?) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
