Re: Metron Installation on an Ambari-Managed Cluster?

[Otto Fowler]

I am looking for a fully scriptable solution, where I don’t have to go into
ambari, such as running the ansible playbook.
Before I would run build and playbook from docker.


On March 24, 2017 at 19:34:12, Matt Foley (ma...@apache.org) wrote:

https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install
is much more recent. It should still work, but I’m not sure how RPM
generation has changed, especially after David’s changes in METRON-671
(Refactor Ansible deployment to use Ambari). Of course the two scenarios
are different (your link is about installing Metron AFTER fully installing
HDP, while this link is about installing it all together), but they use
essentially the same steps.

The above linked article does have a lot of cruft based on working around
previously-existing bugs that have now been fixed, so if something looks
like it’s already been done, or doesn’t make sense, it may be okay to
ignore.

The one thing both articles do that just isn’t necessary, is make you
install/run Docker and do the RPM build on the cluster. I routinely do the
Docker-based RPM build on my Mac, and move only the RPMs to the install
directory used by the Metron MPack for Ambari (/localrepo/) on the Metron
nodes.

So if the Docker RPM build is the main impediment, try doing that.
--Matt


On 3/24/17, 2:03 PM, "Otto Fowler"  wrote:

I have used
https://cwiki.apache.org/confluence/display/METRON/Metron+Installation+on+an+Ambari-Managed+Cluster
as
a guide in the past, but it is out of date,
I don’t think it can possibly work with docker rpm build now.

Does anyone have any ideas what it would take to get this workflow working
again?

[DISCUSS] Kerberos Support

[Casey Stella]

Hi All,

I'd like to talk and start to formulate a plan around supporting running
Metron on a kerberized cluster.  This is a big bundle of work and seems
dauntingly nebulous, so I wanted to have a chat and get a firm direction.
When initially contemplating the issue, it's apparent that there are a few
snags that need to be thought through:

   - The kafka spout (storm-kafka) from apache does not support interacting
   with kerberized kafka
   - We need to ensure the kafka writers are passing along the security
   protocol
   - We need to ensure that the credentials are auto-renewed

Since I knew this was necessary, I wanted to get over this initial barrier
and submitted a couple of PRs associated with the following JIRAs that are
in review now:

   - METRON-793: Migrate to storm-kafka-client, a kafka spout which
   supports kerberized kafka
   - METRON-797: Pass along security protocols and set up autorenewal

Between these, we get what I believe is the enrichment, profiler and parser
topologies and interactions with hbase and hdfs from them working along
with the MR jobs.  We still lack a few things:

   - A document describing the process of kerberizing a vagrant environment
   to enable testing
   - Kerberos support for the librdkafka-based sensors (bro and fastcapa)
   - Kerberos support for the sensors that use the console-producer (snort
   and yaf)
  - This should be as easy as ensuring to pass the right params to the
  console producer, but still, we need to adjust the sensor stubs
to make the
  right call.
   - Kerberos support for the mpack
   - Kerberos support for the REST API
   - Kerberos support for the pycapa

I'm tracking these as JIRAs with the label of "kerberos" (see
https://issues.apache.org/jira/browse/METRON-802?jql=labels%20%3D%20kerberos%20AND%20project%20%3D%20Metron
)

Please chime in if I missed something; I'll add it to the list.

Best,

Casey

[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
Ok, update on the integration tests.  It appears that the issue is that 
storm 1.0.3 does not consistently kill the slots when shutting down.  It times 
out after a minute and dies.  The fix that I am testing is to directly close 
the slot in that case.  I've run the travis build about 8 times with no 
failures.  I'm running locally 20 times before I claim it's fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Metron Installation on an Ambari-Managed Cluster?

[Matt Foley]

https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install
is much more recent.  It should still work, but I’m not sure how RPM generation 
has changed, especially after David’s changes in METRON-671 (Refactor Ansible 
deployment to use Ambari).  Of course the two scenarios are different (your 
link is about installing Metron AFTER fully installing HDP, while this link is 
about installing it all together), but they use essentially the same steps.

The above linked article does have a lot of cruft based on working around 
previously-existing bugs that have now been fixed, so if something looks like 
it’s already been done, or doesn’t make sense, it may be okay to ignore.

The one thing both articles do that just isn’t necessary, is make you 
install/run Docker and do the RPM build on the cluster.  I routinely do the 
Docker-based RPM build on my Mac, and move only the RPMs to the install 
directory used by the Metron MPack for Ambari (/localrepo/) on the Metron nodes.

So if the Docker RPM build is the main impediment, try doing that.
--Matt


On 3/24/17, 2:03 PM, "Otto Fowler"  wrote:

I have used

https://cwiki.apache.org/confluence/display/METRON/Metron+Installation+on+an+Ambari-Managed+Cluster
as
a guide in the past, but it is out of date,
I don’t think it can possibly work with docker rpm build now.

Does anyone have any ideas what it would take to get this workflow working
again?

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107988742
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.scss
 ---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+@import "../../_variables.scss";
+
+.form-title
+{
+  padding-left: 25px;
+}
+
+.title
+{
+  font-size: 12px;
+}
+
+.close-button
+{
+  padding-right: 20px;
+}
+
+.field-schema-row
+{
+  display: table;
+  width: 100%;
+  padding: 10px 0px 10px 0px;
+  background: #383838;
+}
+
+.field-schema-edit-title
+{
+  font-size: 15px;
+  font-weight: bold;
+}
+
+.field-schema-cell
+{
+  float: none;
+  font-size: 13px;
+  height: 15px;
+  line-height: 15px;
+  display: table-cell;
+  vertical-align: middle;
+  word-break: break-all;
+  font-family: Roboto-Regular;
+}
+
+.field-schema-cell-title
+{
+  @extend .field-schema-cell;
+  color: $silver-color;
+  font-size: 14px;
+}
+
+.field-schema-cell-value
+{
+  @extend .field-schema-cell;
+  font-size: 13px;
+}
+
+.small-close-button {
+  font-size:20px;
+  color:#bdbdbd;
+  cursor: pointer;
+}
+
+.config
+{
+  background: $edit-background;
+  border: 1px solid $edit-background-border;
+}
+
+.enrichment-cell
+{
+  display: table-cell;
+  vertical-align: middle;
+  float: none;
+  min-height: 35px;
+  line-height: 20px;
+}
+
+.enrichments-edit-title
+{
+  font-size: 15px;
+  font-weight: bold;
+}
+
+.transform-info-row
+{
+  display: table;
+  width: 100%;
+  min-height: 35px;
+}
+
+.edit-pane-readonly
+{
+  background: #083B44;
--- End diff --

should be in variables


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107987263
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.html
 ---
@@ -0,0 +1,113 @@
+
+  
+
+  Schema
+  
+
+
+  
+  
+
+  
+
+  
+
+
+  
+A data sample is need to view/configure the complete schema.
+  
+
+  
+ Field 
+ Changes 
+  
+
+  
+  
+  
+ {{ 
fieldSchemaRow.outputFieldName }} 
+ 
+
+  
+  
+  
+  
+
+
+  
+
+  
+  
+
+
+
+   {{ 
fieldSchemaRow.outputFieldName }} 
+  
+
+  
+
+
+
+
+  INPUT FIELD
+  
+  
+ {{ fieldSchemaRow.inputFieldName }} 

+  
+
+
+
+
+  NAME
+  
+
+
+
+
+  TRANSFORMATIONS
--- End diff --

for attribute not dynamic, wont' link to the field


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r108007221
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.spec.ts
 ---
@@ -0,0 +1,506 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/* tslint:disable:no-unused-variable */
+/* tslint:disable:max-line-length */
+
+import { TestBed, async, ComponentFixture } from '@angular/core/testing';
+import {Http} from '@angular/http';
+import {SimpleChanges, SimpleChange} from '@angular/core';
+import {SensorParserConfigService} from 
'../../service/sensor-parser-config.service';
+import {StellarService} from '../../service/stellar.service';
+import {MetronAlerts} from '../../shared/metron-alerts';
+import {SensorFieldSchemaModule} from './sensor-field-schema.module';
+import {SensorFieldSchemaComponent, FieldSchemaRow} from 
'./sensor-field-schema.component';
+import {KafkaService} from '../../service/kafka.service';
+import {Observable} from 'rxjs/Observable';
+import {StellarFunctionDescription} from 
'../../model/stellar-function-description';
+import {SensorParserConfig} from '../../model/sensor-parser-config';
+import {SensorEnrichmentConfig, EnrichmentConfig, ThreatIntelConfig} from 
'../../model/sensor-enrichment-config';
+import {ParseMessageRequest} from '../../model/parse-message-request';
+import {AutocompleteOption} from '../../model/autocomplete-option';
+import {FieldTransformer} from '../../model/field-transformer';
+import {SensorEnrichmentConfigService} from 
'../../service/sensor-enrichment-config.service';
+
+
+class MockSensorParserConfigService {
+
+parseMessage(parseMessageRequest: ParseMessageRequest): Observable<{}> 
{
+let parsedJson = {
+'elapsed': 415,
+'code': 200,
+'ip_dst_addr': '207.109.73.154',
+'original_string': '1467011157.401 415 127.0.0.1 TCP_MISS/200 
337891 GET http://www.aliexpress.com/',
+'method': 'GET',
+'bytes': 337891,
+'action': 'TCP_MISS',
+'ip_src_addr': '127.0.0.1',
+'url': 'http://www.aliexpress.com/af/shoes.html?',
+'timestamp': '1467011157.401'
+};
+return Observable.create((observable) => {
+observable.next(parsedJson);
+observable.complete();
+});
+}
+}
+
+class MockTransformationValidationService {
+public listSimpleFunctions(): Observable 
{
+let stellarFunctionDescription: StellarFunctionDescription[] = [];
+stellarFunctionDescription.push(new 
StellarFunctionDescription('TO_LOWER', 'TO_LOWER description', ['input - input 
field']));
+stellarFunctionDescription.push(new 
StellarFunctionDescription('TO_UPPER', 'TO_UPPER description', ['input - input 
field']));
+stellarFunctionDescription.push(new 
StellarFunctionDescription('TRIM', 'Lazy to copy desc', ['input - input 
field']));
+return Observable.create((observer) => {
+observer.next(stellarFunctionDescription);
+observer.complete();
+});
+}
+}
+
+class MockSensorEnrichmentConfigService {
+public getAvailable(): Observable {
+return Observable.create((observer) => {
+observer.next(['geo', 'host', 'whois']);
+observer.complete();
+});
+}
+}
+
+class MockKafkaService {
+
+}
+
+describe('Component: SensorFieldSchema', () => {
+let component: SensorFieldSchemaComponent;
+let sensorEnrichmentConfigService: SensorEnrichmentConfigService;
+let sensorParserConfigService: SensorParserConfigService;
+let fixture: ComponentFixture;
+let transformationValidationService: StellarService;
+
+let 

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r108012411
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.spec.ts
 ---
@@ -0,0 +1,506 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/* tslint:disable:no-unused-variable */
+/* tslint:disable:max-line-length */
+
+import { TestBed, async, ComponentFixture } from '@angular/core/testing';
+import {Http} from '@angular/http';
+import {SimpleChanges, SimpleChange} from '@angular/core';
+import {SensorParserConfigService} from 
'../../service/sensor-parser-config.service';
+import {StellarService} from '../../service/stellar.service';
+import {MetronAlerts} from '../../shared/metron-alerts';
+import {SensorFieldSchemaModule} from './sensor-field-schema.module';
+import {SensorFieldSchemaComponent, FieldSchemaRow} from 
'./sensor-field-schema.component';
+import {KafkaService} from '../../service/kafka.service';
+import {Observable} from 'rxjs/Observable';
+import {StellarFunctionDescription} from 
'../../model/stellar-function-description';
+import {SensorParserConfig} from '../../model/sensor-parser-config';
+import {SensorEnrichmentConfig, EnrichmentConfig, ThreatIntelConfig} from 
'../../model/sensor-enrichment-config';
+import {ParseMessageRequest} from '../../model/parse-message-request';
+import {AutocompleteOption} from '../../model/autocomplete-option';
+import {FieldTransformer} from '../../model/field-transformer';
+import {SensorEnrichmentConfigService} from 
'../../service/sensor-enrichment-config.service';
+
+
+class MockSensorParserConfigService {
+
+parseMessage(parseMessageRequest: ParseMessageRequest): Observable<{}> 
{
+let parsedJson = {
+'elapsed': 415,
+'code': 200,
+'ip_dst_addr': '207.109.73.154',
+'original_string': '1467011157.401 415 127.0.0.1 TCP_MISS/200 
337891 GET http://www.aliexpress.com/',
+'method': 'GET',
+'bytes': 337891,
+'action': 'TCP_MISS',
+'ip_src_addr': '127.0.0.1',
+'url': 'http://www.aliexpress.com/af/shoes.html?',
+'timestamp': '1467011157.401'
+};
+return Observable.create((observable) => {
+observable.next(parsedJson);
+observable.complete();
+});
+}
+}
+
+class MockTransformationValidationService {
+public listSimpleFunctions(): Observable 
{
+let stellarFunctionDescription: StellarFunctionDescription[] = [];
+stellarFunctionDescription.push(new 
StellarFunctionDescription('TO_LOWER', 'TO_LOWER description', ['input - input 
field']));
+stellarFunctionDescription.push(new 
StellarFunctionDescription('TO_UPPER', 'TO_UPPER description', ['input - input 
field']));
+stellarFunctionDescription.push(new 
StellarFunctionDescription('TRIM', 'Lazy to copy desc', ['input - input 
field']));
+return Observable.create((observer) => {
+observer.next(stellarFunctionDescription);
+observer.complete();
+});
+}
+}
+
+class MockSensorEnrichmentConfigService {
+public getAvailable(): Observable {
+return Observable.create((observer) => {
+observer.next(['geo', 'host', 'whois']);
+observer.complete();
+});
+}
+}
+
+class MockKafkaService {
+
+}
+
+describe('Component: SensorFieldSchema', () => {
+let component: SensorFieldSchemaComponent;
+let sensorEnrichmentConfigService: SensorEnrichmentConfigService;
+let sensorParserConfigService: SensorParserConfigService;
+let fixture: ComponentFixture;
+let transformationValidationService: StellarService;
+
+let 

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107987434
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.html
 ---
@@ -0,0 +1,113 @@
+
+  
+
+  Schema
+  
+
+
+  
+  
+
+  
+
+  
+
+
+  
+A data sample is need to view/configure the complete schema.
+  
+
+  
+ Field 
+ Changes 
+  
+
+  
+  
+  
+ {{ 
fieldSchemaRow.outputFieldName }} 
+ 
+
+  
+  
+  
+  
+
+
+  
+
+  
+  
+
+
+
+   {{ 
fieldSchemaRow.outputFieldName }} 
+  
+
+  
+
+
+
+
+  INPUT FIELD
+  
+  
+ {{ fieldSchemaRow.inputFieldName }} 

+  
+
+
+
+
+  NAME
+  
+
+
+
+
+  TRANSFORMATIONS
+   

+   {{ 
fieldSchemaRow.preview }} 
+
+
+
+
+  ENRICHMENTS
+   
+
+
+
+
+  THREAT INTEL 
--- End diff --

Another one: for attribute not linked to field name or id


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107953775
  
--- Diff: metron-interface/metron-config/src/app/login/login.component.html 
---
@@ -0,0 +1,31 @@
+
+
+  
--- End diff --

This should be implemented as an form to benefit from things like default 
submit actions (enter to login) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107987338
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.html
 ---
@@ -0,0 +1,113 @@
+
+  
+
+  Schema
+  
+
+
+  
+  
+
+  
+
+  
+
+
+  
+A data sample is need to view/configure the complete schema.
+  
+
+  
+ Field 
+ Changes 
+  
+
+  
+  
+  
+ {{ 
fieldSchemaRow.outputFieldName }} 
+ 
+
+  
+  
+  
+  
+
+
+  
+
+  
+  
+
+
+
+   {{ 
fieldSchemaRow.outputFieldName }} 
+  
+
+  
+
+
+
+
+  INPUT FIELD
+  
+  
+ {{ fieldSchemaRow.inputFieldName }} 

+  
+
+
+
+
+  NAME
+  
+
+
+
+
+  TRANSFORMATIONS
+   

+   {{ 
fieldSchemaRow.preview }} 
+
+
+
+
+  ENRICHMENTS
--- End diff --

for attr not linking to field


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107956884
  
--- Diff: 
metron-interface/metron-config/src/app/login/login.component.spec.ts ---
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {async, inject, TestBed} from '@angular/core/testing';
+import {AuthenticationService} from '../service/authentication.service';
+import {LoginComponent} from './login.component';
+
+class MockAuthenticationService {
--- End diff --

This mock is used elsewhere too, should it be extracted to an import?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107988671
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.scss
 ---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+@import "../../_variables.scss";
+
+.form-title
+{
+  padding-left: 25px;
+}
+
+.title
+{
+  font-size: 12px;
+}
+
+.close-button
+{
+  padding-right: 20px;
+}
+
+.field-schema-row
+{
+  display: table;
+  width: 100%;
+  padding: 10px 0px 10px 0px;
+  background: #383838;
+}
+
+.field-schema-edit-title
+{
+  font-size: 15px;
+  font-weight: bold;
+}
+
+.field-schema-cell
+{
+  float: none;
+  font-size: 13px;
+  height: 15px;
+  line-height: 15px;
+  display: table-cell;
+  vertical-align: middle;
+  word-break: break-all;
+  font-family: Roboto-Regular;
+}
+
+.field-schema-cell-title
+{
+  @extend .field-schema-cell;
+  color: $silver-color;
+  font-size: 14px;
+}
+
+.field-schema-cell-value
+{
+  @extend .field-schema-cell;
+  font-size: 13px;
+}
+
+.small-close-button {
+  font-size:20px;
+  color:#bdbdbd;
--- End diff --

should be a variable


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107954344
  
--- Diff: metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec 
---
@@ -313,6 +315,25 @@ This package installs the Metron Profiler 
%{metron_home}
 %{metron_home}/flux/profiler/remote.yaml
 %attr(0644,root,root) 
%{metron_home}/lib/metron-profiler-%{full_version}-uber.jar
 
+# 
~~
+
+%packageconfig
+Summary:Metron Config
--- End diff --

Should be more descriptive. e.g.: Metron Configuration UI web application


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107957813
  
--- Diff: 
metron-interface/metron-config/src/app/model/parse-message-request.ts ---
@@ -0,0 +1,23 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {SensorParserConfig} from './sensor-parser-config';
+export class ParseMessageRequest {
--- End diff --

Should the be renamed to GrokParseMessage and have a separate message type 
for other parsers?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107977492
  
--- Diff: 
metron-interface/metron-config/src/app/model/threat-triage-config.ts ---
@@ -0,0 +1,23 @@
+import {RiskLevelRule} from './risk-level-rule';
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+export class ThreatTriageConfig {
+  riskLevelRules: RiskLevelRule[] = [];
+  aggregator: string = 'MAX';
--- End diff --

Should be an enum


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107988841
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.scss
 ---
@@ -0,0 +1,168 @@
+/**
--- End diff --

several colors in here which should probably be extracted as variables


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107977639
  
--- Diff: metron-interface/metron-config/src/app/model/topology-status.ts 
---
@@ -0,0 +1,26 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+export class TopologyStatus {
--- End diff --

Numeric values are forced to strings here. Should this not be done on 
display in the view?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Metron Installation on an Ambari-Managed Cluster?

[Otto Fowler]

I have used
https://cwiki.apache.org/confluence/display/METRON/Metron+Installation+on+an+Ambari-Managed+Cluster
as
a guide in the past, but it is out of date,
I don’t think it can possibly work with docker rpm build now.

Does anyone have any ideas what it would take to get this workflow working
again?

[GitHub] incubator-metron issue #488: METRON-796: Mpack uses wrong group for owning H...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/488
  
+1 then, thanks.  Please expand METRON-349 to include these issues too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #490: DO NOT MERGE METRON-797: Pass security.protocol...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/490
  
Thanks, there didn't seem to be a jira for parsers, so I expanded the scope 
of METRON-798 and made bro it's first sub-task.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #488: METRON-796: Mpack uses wrong group for owning H...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/incubator-metron/pull/488
  
I agree, let's get this sorted, and then follow up with a general review of 
the permissions once we've got the rest of the security pieces in place.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #488: METRON-796: Mpack uses wrong group for owning H...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/488
  
@mattf-horton Permissions on the other items are generally 775, so they can 
be read as needed (and should be scaled back once we have everything lined up 
with the user as @simonellistonball mentions). I just wanted to touch things as 
little as possible to get them back in a working state.

We should either expand out or replace METRON-349 (including closing off 
permissions) as the actual solution to the problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #488: METRON-796: Mpack uses wrong group for owning H...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/488
  
Thanks.  Looks okay to me now, for the intended purpose.  +0 due to Simon's 
concerns; I'd prefer someone with more experience in Metron give you the plus 
one.

Regarding @simonellistonball 's comments:
The prior assignment of group ownership as `metron_group` in 
enrichment_commands.py, line 114, and indexing_commands.py, line 104, dates 
from the commit for 
* [METRON-671 Refactor existing Ansible deployment to use Ambari MPack 
(dlyle via 
justinleet)#436](https://github.com/justinleet/incubator-metron/commit/68a334a8717d2a6b82f7f9651c57bc75e256ba97)
Prior to PR#436 they were both set to `user_group`.

@dlyle65535 are you okay with setting these back to `user_group`, as they 
were before?
ie do you agree with @justinleet 's reasoning?  Thanks.

@justinleet , you say: "Other items, such as configs and installation 
files, were just left as the metron group."  Why is this okay?  Don't the 
topologies have to read config files too?  Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #490: DO NOT MERGE METRON-797: Pass security.protocol...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/490
  
Good catch.  I made the dependency explicit and created the JIRAs that I 
know about for kerberos 
[here](https://issues.apache.org/jira/browse/METRON-802?jql=labels%20%3D%20kerberos%20AND%20project%20%3D%20Metron)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #488: METRON-796: Mpack uses wrong group for o...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/488#discussion_r107965896
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
 ---
@@ -39,7 +39,7 @@
 tmp_dir = Script.get_tmp_dir()
 
 hostname = config['hostname']
-metron_group = config['configurations']['cluster-env']['metron_group']
+hadoop_group = config['configurations']['cluster-env']['user_group']
--- End diff --

Updated with a comment to clarify things a bit.  Let me know if you think 
there's anything else we want to add or clarify.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #488: METRON-796: Mpack uses wrong group for o...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/488#discussion_r107964537
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
 ---
@@ -39,7 +39,7 @@
 tmp_dir = Script.get_tmp_dir()
 
 hostname = config['hostname']
-metron_group = config['configurations']['cluster-env']['metron_group']
+hadoop_group = config['configurations']['cluster-env']['user_group']
--- End diff --

Yuck (but thanks for the data).  That's kinda perverse, but since it's in 
Ambari it's out of our scope to fix.  So I agree with your current usage.  A 
comment would be nice, since 
a. it's the `hadoop` group not the `users` group; and furthermore
b. its members are the *stack process user ids* not individual users.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #490: DO NOT MERGE METRON-797: Pass security.protocol...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/490
  
@cestella , In the Jira, please add the dependencies as blockers (at least 
METRON-793), and create (if it doesn't exist yet) the jira(s) for the needed 
follow-on work and link to them too.  Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #488: METRON-796: Mpack uses wrong group for o...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/488#discussion_r107957631
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
 ---
@@ -39,7 +39,7 @@
 tmp_dir = Script.get_tmp_dir()
 
 hostname = config['hostname']
-metron_group = config['configurations']['cluster-env']['metron_group']
+hadoop_group = config['configurations']['cluster-env']['user_group']
--- End diff --

please put this line in the `hadoop params` section (circa line 80).

Also, why equate hadoop_group to 'user_group'?  In typical hadoop 
deployments there is already a group named 'hadoop'.  It is managed by Ambari.  
In a recent test install of HDP+Metron, it was defined as follows (from 
/etc/group), which I think would meet your intended use since both storm and 
metron are included:
* hadoop: x 
:1004:hive,storm,zookeeper,infra-solr,zeppelin,metron,livy,elasticsearch,spark,flume,kafka,hdfs,yarn,kibana,mapred,hbase,hcat



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
@justinleet It is unclear and that test has had some intermittent issues, 
especially under load, I've noticed.  I added a more descriptive message to 
help diagnose.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
@cestella Just noticed Travis after I commented.  I'm moderately surprised 
that the most recent PR would break it, do you know what the issue is?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
+1, was able to follow Mike's instructions, with a couple caveats.

- Group authorization command was missing
```
/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=node1:2181 --add --allow-principal User:storm-metron_cluster 
--allow-principal User:justin --group jsonMap_parser
```
- Topic authorization command on the enrichments topic side was missing.
```
 /usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=node1:2181 --add --allow-principal User:storm-metron_cluster 
--allow-principal User:justin --topic enrichments
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932800
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.ts
 ---
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { Component, OnInit, ViewChild } from '@angular/core';
+import {GlobalConfigService} from '../service/global-config.service';
+import {MetronAlerts} from '../shared/metron-alerts';
+import {MetronDialogBox} from '../shared/metron-dialog-box';
+
+@Component({
+  selector: 'metron-config-general-settings',
+  templateUrl: './general-settings.component.html',
+  styleUrls: ['./general-settings.component.scss']
+})
+export class GeneralSettingsComponent implements OnInit {
+
+  @ViewChild('globalSettingsForm') globalSettingsForm: any;
--- End diff --

Type should be limited to the child component type


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932817
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.ts
 ---
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { Component, OnInit, ViewChild } from '@angular/core';
+import {GlobalConfigService} from '../service/global-config.service';
+import {MetronAlerts} from '../shared/metron-alerts';
+import {MetronDialogBox} from '../shared/metron-dialog-box';
+
+@Component({
+  selector: 'metron-config-general-settings',
+  templateUrl: './general-settings.component.html',
+  styleUrls: ['./general-settings.component.scss']
+})
+export class GeneralSettingsComponent implements OnInit {
+
+  @ViewChild('globalSettingsForm') globalSettingsForm: any;
+
+  globalConfig: {} = {
+'es.date.format': '-'
+  };
+
+  private fieldValidations: string;
+
+  constructor(private globalConfigService: GlobalConfigService, private 
metronAlerts: MetronAlerts,
--- End diff --

Not sure why this class is serialising to JSON for fieldValidations. Feels 
like unnecessary overhead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932789
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.spec.ts
 ---
@@ -0,0 +1,161 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Inject} from '@angular/core';
+import {async, TestBed, ComponentFixture} from '@angular/core/testing';
+import {Http} from '@angular/http';
+import {GeneralSettingsComponent} from './general-settings.component';
+import {MetronAlerts} from '../shared/metron-alerts';
+import {MetronDialogBox} from '../shared/metron-dialog-box';
+import {GlobalConfigService} from '../service/global-config.service';
+import {GeneralSettingsModule} from './general-settings.module';
+import {Observable} from 'rxjs/Observable';
+import {APP_CONFIG, METRON_REST_CONFIG} from '../app.config';
+import {IAppConfig} from '../app.config.interface';
+
+class MockGlobalConfigService extends GlobalConfigService {
+  _config: any = {};
+  _postSuccess: boolean = true;
+
+  constructor(private http2: Http, @Inject(APP_CONFIG) private config2: 
IAppConfig) {
+super(http2, config2);
+  }
+
+  public post(globalConfig: {}): Observable<{}> {
+if (this._postSuccess) {
+  return Observable.create(observer => {
+observer.next(globalConfig);
+observer.complete();
+  });
+}
+
+return Observable.throw('Error');
+  }
+
+  public get(): Observable<{}> {
+return Observable.create(observer => {
+  observer.next(this._config);
+  observer.complete();
+});
+  }
+}
+
+describe('GeneralSettingsComponent', () => {
+
+  let metronAlerts: MetronAlerts;
+  let metronDialogBox: MetronDialogBox;
+  let component: GeneralSettingsComponent;
+  let globalConfigService: MockGlobalConfigService;
+  let fixture: ComponentFixture;
+  let config = {
+'solr.collection': 'metron',
+'storm.indexingWorkers': 1,
+'storm.indexingExecutors': 2,
+'hdfs.boltBatchSize': 5000,
+'hdfs.boltFieldDelimiter': '|',
+'hdfs.boltFileRotationSize': 5,
+'hdfs.boltCompressionCodecClass': 
'org.apache.hadoop.io.compress.SnappyCodec',
+'hdfs.indexOutput': '/tmp/metron/enriched',
+'kafkaWriter.topic': 'outputTopic',
+'kafkaWriter.keySerializer': 
'org.apache.kafka.common.serialization.StringSerializer',
+'kafkaWriter.valueSerializer': 
'org.apache.kafka.common.serialization.StringSerializer',
+'kafkaWriter.requestRequiredAcks': 1,
+'solrWriter.indexName': 'alfaalfa',
+'solrWriter.shards': 1,
+'solrWriter.replicationFactor': 1,
+'solrWriter.batchSize': 50,
+'fieldValidations': {'field': 'validation'}
+  };
+
+  beforeEach(async(() => {
+
+TestBed.configureTestingModule({
+  imports: [GeneralSettingsModule],
+  providers: [
+{provide: Http},
+MetronAlerts,
+MetronDialogBox,
+{provide: GlobalConfigService, useClass: MockGlobalConfigService},
+{provide: APP_CONFIG, useValue: METRON_REST_CONFIG}
+  ]
+}).compileComponents()
+  .then(() => {
+fixture = TestBed.createComponent(GeneralSettingsComponent);
+component = fixture.componentInstance;
+globalConfigService = 
fixture.debugElement.injector.get(GlobalConfigService);
+metronAlerts = fixture.debugElement.injector.get(MetronAlerts);
+metronDialogBox = 
fixture.debugElement.injector.get(MetronDialogBox);
+  });
+
+  }));
+
+  it('can instantiate GeneralSettingsComponent', async(() => {
+expect(component instanceof GeneralSettingsComponent).toBe(true);
+  }));
+
+  it('should load global config', async(() => {
+globalConfigService._config = config;
+component.ngOnInit();
+
 

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932760
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.html
 ---
@@ -0,0 +1,107 @@
+
+
+  
+
+   General Settings 
+
+
+
+  
+
+  
+  Elasticsearch  
+
+  INDEX DATE FORMAT
+  
+
+
+  
+
+  
+  Validation  
+
+
+  FIELD
+  
+
+
+  
+
+
+
+
+  
+
+
+  
+
+  
--- End diff --

better handled with styling css padding rather than linebreaks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932746
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.html
 ---
@@ -0,0 +1,107 @@
+
+
+  
+
+   General Settings 
+
+
+
+  
+
+  
+  Elasticsearch  
+
+  INDEX DATE FORMAT
+  
+
+
+  
+
+  
+  Validation  
+
+
+  FIELD
+  
+
+
+  
+
+
+
+

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932680
  
--- Diff: metron-interface/metron-config/package.json ---
@@ -0,0 +1,65 @@
+{
+  "name": "clitest",
+  "version": "0.0.0",
--- End diff --

Should track metron version


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932702
  
--- Diff: metron-interface/metron-config/src/app/_main.scss ---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/* cyrillic-ext */
--- End diff --

What are the cyrillic extension comments used for? Also, do we use all the 
weights on Roboto, or can we slim down our app size here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932776
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.spec.ts
 ---
@@ -0,0 +1,161 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Inject} from '@angular/core';
+import {async, TestBed, ComponentFixture} from '@angular/core/testing';
+import {Http} from '@angular/http';
+import {GeneralSettingsComponent} from './general-settings.component';
+import {MetronAlerts} from '../shared/metron-alerts';
+import {MetronDialogBox} from '../shared/metron-dialog-box';
+import {GlobalConfigService} from '../service/global-config.service';
+import {GeneralSettingsModule} from './general-settings.module';
+import {Observable} from 'rxjs/Observable';
+import {APP_CONFIG, METRON_REST_CONFIG} from '../app.config';
+import {IAppConfig} from '../app.config.interface';
+
+class MockGlobalConfigService extends GlobalConfigService {
+  _config: any = {};
+  _postSuccess: boolean = true;
+
+  constructor(private http2: Http, @Inject(APP_CONFIG) private config2: 
IAppConfig) {
+super(http2, config2);
+  }
+
+  public post(globalConfig: {}): Observable<{}> {
+if (this._postSuccess) {
+  return Observable.create(observer => {
+observer.next(globalConfig);
+observer.complete();
+  });
+}
+
+return Observable.throw('Error');
+  }
+
+  public get(): Observable<{}> {
+return Observable.create(observer => {
+  observer.next(this._config);
+  observer.complete();
+});
+  }
+}
+
+describe('GeneralSettingsComponent', () => {
+
+  let metronAlerts: MetronAlerts;
+  let metronDialogBox: MetronDialogBox;
+  let component: GeneralSettingsComponent;
+  let globalConfigService: MockGlobalConfigService;
+  let fixture: ComponentFixture;
+  let config = {
+'solr.collection': 'metron',
+'storm.indexingWorkers': 1,
+'storm.indexingExecutors': 2,
+'hdfs.boltBatchSize': 5000,
+'hdfs.boltFieldDelimiter': '|',
+'hdfs.boltFileRotationSize': 5,
+'hdfs.boltCompressionCodecClass': 
'org.apache.hadoop.io.compress.SnappyCodec',
+'hdfs.indexOutput': '/tmp/metron/enriched',
+'kafkaWriter.topic': 'outputTopic',
+'kafkaWriter.keySerializer': 
'org.apache.kafka.common.serialization.StringSerializer',
+'kafkaWriter.valueSerializer': 
'org.apache.kafka.common.serialization.StringSerializer',
+'kafkaWriter.requestRequiredAcks': 1,
+'solrWriter.indexName': 'alfaalfa',
+'solrWriter.shards': 1,
+'solrWriter.replicationFactor': 1,
+'solrWriter.batchSize': 50,
+'fieldValidations': {'field': 'validation'}
+  };
+
+  beforeEach(async(() => {
+
+TestBed.configureTestingModule({
+  imports: [GeneralSettingsModule],
+  providers: [
+{provide: Http},
+MetronAlerts,
+MetronDialogBox,
+{provide: GlobalConfigService, useClass: MockGlobalConfigService},
+{provide: APP_CONFIG, useValue: METRON_REST_CONFIG}
+  ]
+}).compileComponents()
+  .then(() => {
+fixture = TestBed.createComponent(GeneralSettingsComponent);
+component = fixture.componentInstance;
+globalConfigService = 
fixture.debugElement.injector.get(GlobalConfigService);
+metronAlerts = fixture.debugElement.injector.get(MetronAlerts);
+metronDialogBox = 
fixture.debugElement.injector.get(MetronDialogBox);
+  });
+
+  }));
+
+  it('can instantiate GeneralSettingsComponent', async(() => {
+expect(component instanceof GeneralSettingsComponent).toBe(true);
+  }));
+
+  it('should load global config', async(() => {
+globalConfigService._config = config;
+component.ngOnInit();
+
 

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932715
  
--- Diff: metron-interface/metron-config/src/app/app.component.ts ---
@@ -0,0 +1,41 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Component} from '@angular/core';
+import './rxjs-operators';
--- End diff --

unused import?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932735
  
--- Diff: metron-interface/metron-config/src/app/app.component.ts ---
@@ -0,0 +1,41 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Component} from '@angular/core';
+import './rxjs-operators';
+import {AuthenticationService} from './service/authentication.service';
+import 'jquery';
--- End diff --

unused import? and $ declaration? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932690
  
--- Diff: metron-interface/metron-config/scripts/start_management_ui.sh ---
@@ -0,0 +1,27 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+SCRIPTS_ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+export METRON_REST_URL=${METRON_REST_URL:-http://localhost:8080}
+export MANAGEMENT_UI_PORT=${MANAGEMENT_UI_PORT:-4200}
+npm version > /dev/null
+if [ $? -eq 0 ]; then
+npm install http-server
--- End diff --

should this be in packages.json instead to avoid the need for npm on 
production nodes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932655
  
--- Diff: 
metron-interface/metron-config/e2e/sensor-config/sensor-config.po.ts ---
@@ -0,0 +1,243 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { browser, element, by, protractor } from 'protractor/globals';
+import {changeURL, waitForElementVisibility, waitForElementInVisibility, 
waitForElementPresence} from '../utils/e2e_util';
+
+export class SensorConfigPage {
+
+  _getEnrichmentsMultipleInput() {
--- End diff --

_ usually indicated a private function, should this also be declared 
private?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932605
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.html
 ---
@@ -0,0 +1,106 @@
+
+
+
+
+  
+  
+{{selectedSensorName}}
+  
+
+  
+
+  
+
+
+  
+  
+
+
+
+
+
+  
+{{ item.value }}
+  
+
+  
+{{ item.label 
}}
+{{ 
sensorParserConfigHistory[item.value] ? sensorParserConfigHistory[item.value] : 
"-" }}
+{{ kafkaTopic[item.value] ? kafkaTopic[item.value] : "-" 
}}
+{{ topologyStatus[item.value] ? 
topologyStatus[item.value] : "-" }}
+
+  Grok Statement
+  
+  
+  show more
+
+
+
+
+  
+Transforms
+
+  
+{{ results }}
+{{ 
transformsConfigMap[results] }}
+  
+
+  
+
+  
+Transforms
+
+  
+  show more
+
+
+  AGGREGATOR{{sensorEnrichmentConfig.threatIntel.triageConfig.aggregator}}
+  
+
+  NAMESCORE
+  
+{{ 
getDisplayName(riskLevelRule) }}
+{{ 
riskLevelRule.score }}
+  
+
+  
+  {{ 
getRuleDisplayName() }}
+  show more
+
+  
+
+  
+
+
+  
+  
+EDIT
+
+START
+STOP
--- End diff --

controller logic?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932661
  
--- Diff: metron-interface/metron-config/e2e/sensor-list/sensor-list.po.ts 
---
@@ -0,0 +1,240 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions andx
+ * limitations under the License.
+ */
+
+import { browser, element, by, protractor } from 'protractor/globals';
+import { waitForElementPresence, waitForStalenessOf } from 
'../utils/e2e_util';
+var Promise = require('bluebird');
+
+export class SensorListPage {
+
+clickOnActionsAndWait(parserNames: string[], clickOnClassName: string, 
waitOnClassName: string) {
+let queueForClick = Promise.resolve();
+parserNames.forEach((name) => {
+queueForClick = queueForClick.then((result) => {
+return this.getIconButton(name, clickOnClassName).click();
+});
+});
+
+return queueForClick.then(() => {
+let promiseArray = [];
+parserNames.map(name => {
+
promiseArray.push(this.waitForElement(this.getIconButton(name, 
waitOnClassName)));
+});
+
+return protractor.promise.all(promiseArray).then(args => {
+return args;
+});
+});
+}
+
+clickOnDropdownAndWait(parserNames: string[], dropDownLinkName: 
string, waitOnClassName: string) {
+return protractor.promise.all([this.toggleSelectAll(), 
this.toggleDropdown()]).then(() => {
+
+return element(by.css('span[data-action=\"'+ dropDownLinkName 
+'\"]')).click().then(() => {
+let promiseArray = [];
+parserNames.map(name => {
+
promiseArray.push(this.waitForElement(this.getIconButton(name, 
waitOnClassName)));
--- End diff --

more efficient to avoid side-effects here, since map returns the result of 
a transformation, e.g. promiseArray = parserNames.map( (name) => x ) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932671
  
--- Diff: metron-interface/metron-config/e2e/sensor-list/sensor-list.po.ts 
---
@@ -0,0 +1,240 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions andx
+ * limitations under the License.
+ */
+
+import { browser, element, by, protractor } from 'protractor/globals';
+import { waitForElementPresence, waitForStalenessOf } from 
'../utils/e2e_util';
+var Promise = require('bluebird');
+
+export class SensorListPage {
+
+clickOnActionsAndWait(parserNames: string[], clickOnClassName: string, 
waitOnClassName: string) {
+let queueForClick = Promise.resolve();
+parserNames.forEach((name) => {
+queueForClick = queueForClick.then((result) => {
+return this.getIconButton(name, clickOnClassName).click();
+});
+});
+
+return queueForClick.then(() => {
+let promiseArray = [];
+parserNames.map(name => {
+
promiseArray.push(this.waitForElement(this.getIconButton(name, 
waitOnClassName)));
+});
+
+return protractor.promise.all(promiseArray).then(args => {
+return args;
+});
+});
+}
+
+clickOnDropdownAndWait(parserNames: string[], dropDownLinkName: 
string, waitOnClassName: string) {
+return protractor.promise.all([this.toggleSelectAll(), 
this.toggleDropdown()]).then(() => {
+
+return element(by.css('span[data-action=\"'+ dropDownLinkName 
+'\"]')).click().then(() => {
+let promiseArray = [];
+parserNames.map(name => {
+
promiseArray.push(this.waitForElement(this.getIconButton(name, 
waitOnClassName)));
+});
+
+return protractor.promise.all(promiseArray).then(args => {
+return this.toggleSelectAll().then(() => {
+return args;
+})
+});
+});
+});
+}
+
+closePane(className: string ='.close-button') {
+return this.waitForElement(element(by.css(className))).then(() => {
+return element(by.css(className)).click();
+});
+}
+
+disableParsers(names: string[]) {
+return this.clickOnActionsAndWait(names, 'i.fa-ban', 
'i.fa-check-circle-o');
+}
+
+disableParsersFromDropdown(names: string[]) {
+return this.clickOnDropdownAndWait(names, 'Disable', 
'i.fa-check-circle-o');
+}
+
+deleteParser(name: string) {
+return this.getIconButton(name, '.fa-trash-o').click().then(() => {
+browser.sleep(1000);
+return element(by.css('.metron-dialog 
.btn-primary')).click().then(() => {
--- End diff --

promise chaining 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932638
  
--- Diff: 
metron-interface/metron-config/e2e/sensor-config-readonly/sensor-config-readonly.po.ts
 ---
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { browser, element, by, protractor } from 'protractor/globals';
+import {changeURL, waitForElementVisibility} from '../utils/e2e_util';
+
+export class SensorDetailsPage {
+
+private enableButton;
+private disableButton;
+private startButton;
+private stopButton;
+
+constructor() {
+this.enableButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'ENABLE'));
+this.disableButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'DISABLE'));
+this.startButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'START'));
+this.stopButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'STOP'));
+}
+
+clickToggleShowMoreLess(text: string, index: number) {
+return element.all(by.linkText(text)).get(index).click().then(() 
=> {
+browser.sleep(1000);
+return true;
+})
+}
+
+closePane(name: string) {
+return element(by.css('metron-config-sensor-parser-readonly 
.fa-times')).click().then(() => {
+return true;
+});
+}
+
+disableParser() {
+return waitForElementVisibility(this.disableButton).then(() => {
+return this.disableButton.click().then(() => {
+return waitForElementVisibility(this.enableButton).then(() 
=> {
--- End diff --

promises should be really chained rather than nested... see other instances 
in this class too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932625
  
--- Diff: metron-interface/metron-config/angular-cli.json ---
@@ -0,0 +1,51 @@
+{
+  "project": {
+"version": "1.0.0-beta.15",
--- End diff --

Should we version this to match Metron versions? It's likely to run lock 
step with Metron from a release perspective.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107932616
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.html
 ---
@@ -0,0 +1,106 @@
+
+
+
+
+  
+  
+{{selectedSensorName}}
+  
+
+  
+
+  
+
+
+  
+  
+
+
+
+
+
+  
+{{ item.value }}
+  
+
+  
+{{ item.label 
}}
+{{ 
sensorParserConfigHistory[item.value] ? sensorParserConfigHistory[item.value] : 
"-" }}
+{{ kafkaTopic[item.value] ? kafkaTopic[item.value] : "-" 
}}
+{{ topologyStatus[item.value] ? 
topologyStatus[item.value] : "-" }}
+
+  Grok Statement
+  
+  
+  show more
+
+
+
+
+  
+Transforms
+
+  
+{{ results }}
+{{ 
transformsConfigMap[results] }}
+  
+
+  
+
+  
+Transforms
+
+  
+  show more
+
+
+  AGGREGATOR{{sensorEnrichmentConfig.threatIntel.triageConfig.aggregator}}
+  
+
+  NAMESCORE
+  
+{{ 
getDisplayName(riskLevelRule) }}
+{{ 
riskLevelRule.score }}
+  
+
+  
+  {{ 
getRuleDisplayName() }}
+  show more
--- End diff --

should this be encapsulated in a controller function?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/489#discussion_r107931875
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-grok/sensor-grok.component.html
 ---
@@ -0,0 +1,42 @@
+
+
+
+Grok Validator
+
+
+
+
+STATEMENT
+ 

+
+
+TEST
--- End diff --

should this logic be in controller?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #490: DO NOT MERGE METRON-797: Pass security.protocol...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/490
  
# Testing Plan

## Preliminaries

* Please perform the following tests on the `full-dev` vagrant environment.
* Set an environment variable to indicate `METRON_HOME`:
`export METRON_HOME=/usr/metron/0.3.1` 


## Ensure Data Flows from the Indices
Ensure that with a basic full-dev we get data into the elasticsearch
indices and into HDFS.

## (Optional) Free Up Space on the virtual machine

First, let's free up some headroom on the virtual machine.  If you are 
running this on a
multinode cluster, you would not have to do this.
* Stop and disable Metron in Ambari
* Kill monit via `service monit stop`
* Kill tcpreplay via `for i in $(ps -ef | grep tcpreplay | awk '{print 
$2}');do kill -9 $i;done`
* Kill yaf via `for i in $(ps -ef | grep yaf | awk '{print $2}');do kill -9 
$i;done`
* Kill bro via `for i in $(ps -ef | grep bro | awk '{print $2}');do kill -9 
$i;done`

## Test the PCAP topology

A new kafka spout necessitates testing pcap.
### Install and start pycapa 
```
# set env vars
export PYCAPA_HOME=/opt/pycapa
export PYTHON27_HOME=/opt/rh/python27/root

# Install these packages via yum (RHEL, CentOS)
yum -y install epel-release centos-release-scl 
yum -y install "@Development tools" python27 python27-scldevel 
python27-python-virtualenv libpcap-devel libselinux-python

# Setup directories
mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME

#Grab pycapa from git 
cd ~
git clone https://github.com/apache/incubator-metron.git
cp -R ~/incubator-metron/metron-sensors/pycapa* $PYCAPA_HOME

# Create virtualenv
export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv

# Build it
cd ${PYCAPA_HOME}/pycapa
# activate the virtualenv
source ${PYCAPA_HOME}/pycapa-venv/bin/activate
pip install -r requirements.txt
python setup.py install

# Run it
cd ${PYCAPA_HOME}/pycapa-venv/bin
pycapa --producer --topic pcap -i eth1 -k node1:6667
```
### Ensure pycapa can write to HDFS
* Ensure that `/apps/metron/pcap` exists and can be written to by the
  storm user.  If not, then:
```
sudo su - hdfs
hadoop fs -mkdir -p /apps/metron/pcap
hadoop fs -chown metron:hadoop /apps/metron/pcap
hadoop fs -chmod 775 /apps/metron/pcap
exit
``` 
* Start the pcap topology via `$METRON_HOME/bin/start_pcap_topology.sh`
* Watch the topology in the Storm UI and kill the packet capture utility 
from before, when the number of packets ingested is over 3k.  Ensure that at at 
least 3 files exist on HDFS by running `hadoop fs -ls /apps/metron/pcap`
* Choose a file (denoted by $FILE) and dump a few of the contents using the 
pcap_inspector utility via `$METRON_HOME/bin/pcap_inspector.sh -i $FILE -n 5`
* Choose one of the lines and note the `ip_dst_port`.
  * Note that when you run the commands below, the resulting file will be 
placed in the execution directory where you kicked off the job from.
* Run a Stellar query filter query by executing a command similar to the 
following, with the values noted above (match your start_time format to the 
date format provided - default is to use millis since epoch):
```
$METRON_HOME/bin/pcap_query.sh query -st "20160617" -df "MMdd" -query 
"ip_dst_port == 22" -rpf 500
```
  * Note that if your MR job fails because of a lack of user directory for 
`root`, then the following will create the directory appropriately:
```
sudo su - hdfs
hadoop fs -mkdir /user/root
hadoop fs -chown root:hadoop /user/root
hadoop fs -chmod 755 /user/root
exit
```
* Verify the MR job finishes successfully. Upon completion, you should see 
multiple files named with relatively current datestamps in your current 
directory, e.g. pcap-data-20160617160549737+.pcap
* Copy the files to your local machine and verify you can them it in 
Wireshark. Open the files and ensure that they contain only packets to the 
destination port in question. 

## Test the Profiler

### Setup
* Ensure that Metron is stopped and put in maintenance mode in Ambari
* Create the profiler hbase table
`echo "create 'profiler', 'P'" | hbase shell`

* Open `~/rand_gen.py` and paste the following:
```
#!/usr/bin/python
import random
import sys
import time
def main():
  mu = float(sys.argv[1])
  sigma = float(sys.argv[2])
  freq_s = int(sys.argv[3])
  while True:
out = '{ "value" : ' + str(random.gauss(mu, sigma)) + ' }'
print out
sys.stdout.flush()
time.sleep(freq_s)

if __name__ == '__main__':
  main()
```
This will generate random JSON maps with a 

[GitHub] incubator-metron pull request #490: DO NOT MERGE METRON-797: Pass security.p...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/490


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #490: DO NOT MERGE METRON-797: Pass security.p...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/490

DO NOT MERGE METRON-797: Pass security.protocol and enable auto-renew for 
the storm topologies

## Contributor Comments

METRON-793 migrated the storm topologies to the storm-kafka-client spout, 
which supports kerberos. To complete the kerberos work on the existing 
topologies, we need to be able to enable the spouts and kafka writers to use 
security protocols other than PLAINTEXT. Also, enabling auto-renew plugins for 
storm will enable the topologies to run for extended durations in a kerberized 
cluster.
NOTE: This does not encompass MPack changes to enable kerberos or fix the 
sensors to work with a kerberized kafka. That would be follow-on work.

NOTE: This depends on METRON-793.  As such, this PR includes it, so do not 
merge until 793 is merged.

An acceptance test plan will follow as well as instructions for kerberizing 
vagrant and running the topologies in said environment.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-797

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #490


commit 1a0dd8dafd7e722e9421cebe2f19fe540ccea73d
Author: cstella 
Date:   2017-03-22T14:28:53Z

METRON-793: Migrate to storm-kafka-client kafka spout from storm-kafka

commit 9e2ff8aa7e6113d5aba89ba1c1b5eebb375ffd6d
Author: cstella 
Date:   2017-03-22T14:35:48Z

Updating to remove commented out code.

commit 8a5100b5d9a3abdc464899055de3c0c86e4487d9
Author: cstella 
Date:   2017-03-22T18:06:39Z

Better documentation

commit b6adf40ed7c5ce696d4e9ee7871fcd8bbdca9923
Author: cstella 
Date:   2017-03-22T18:42:23Z

Adding for the spout configuration.

commit 49ccefb5435e6b5057fbba8c5b33f532dbad8904
Author: cstella 

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107915490
  
--- Diff: 
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.storm.kafka.flux;
+
+import com.google.common.base.Joiner;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.metron.common.utils.KafkaUtils;
+import org.apache.storm.kafka.spout.*;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.OutputFieldsGetter;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/**
+ * This is a convenience layer on top of the KafkaSpoutConfig.Builder 
available in storm-kafka-client.
+ * The justification for this class is two-fold.  First, there are a lot 
of moving parts and a simplified
+ * approach to constructing spouts is useful.  Secondly, and perhaps more 
importantly, the Builder pattern
+ * is decidedly unfriendly to use inside of Flux.  Finally, we can make 
things a bit more friendly by only requiring
+ * zookeeper and automatically figuring out the brokers for the bootstrap 
server.
+ *
+ * @param  The kafka key type
+ * @param  The kafka value type
+ */
+public class SimpleStormKafkaBuilder extends 
KafkaSpoutConfig.Builder {
+  final static String STREAM = "default";
+
+  /**
+   * The fields exposed by the kafka consumer.  These will show up in the 
Storm tuple.
+   */
+  public enum FieldsConfiguration {
+KEY("key", record -> record.key()),
+VALUE("value", record -> record.value()),
+PARTITION("partition", record -> record.partition()),
+TOPIC("topic", record -> record.topic())
+;
+String fieldName;
+Function recordExtractor;
+
+FieldsConfiguration(String fieldName, Function 
recordExtractor) {
+  this.recordExtractor = recordExtractor;
+  this.fieldName = fieldName;
+}
+
+/**
+ * Return a list of the enums
+ * @param configs
+ * @return
+ */
+public static List toList(String... configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Return a list of the enums from their string representation.
+ * @param configs
+ * @return
+ */
+public static List toList(List configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Construct a Fields object from an iterable of enums.  These fields 
are the fields
+ * exposed in the Storm tuple emitted from the spout.
+ * @param configs
+ * @return
+ */
+public static Fields getFields(Iterable configs) {
+  List fields = new ArrayList<>();
+  for(FieldsConfiguration config : configs) {
+fields.add(config.fieldName);
+  }
+  return new Fields(fields);
+}
+  }
+
+  /**
+   * Build a tuple given the fields and the topic.  We want to use our 
FieldsConfiguration enum
+   * to define what this tuple looks like.
+   * @param  The key type in kafka
+   * @param  The value 

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107915471
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
 ---
@@ -106,19 +105,22 @@ public static TopologyBuilder build(String 
zookeeperUrl,
   /**
* Create a spout that consumes tuples from a Kafka topic.
*
-   * @param zookeeperUrlZookeeper URL
+   * @param zkQuorum Zookeeper URL
* @param sensorType  Type of sensor
-   * @param offset  Kafka topic offset where the topology 
will start; BEGINNING, END, WHERE_I_LEFT_OFF
-   * @param kafkaSpoutConfigOptions Configuration options for the kafka 
spout
+   * @param kafkaConfigOptional Configuration options for the kafka 
spout
* @param parserConfigConfiguration for the parser
* @return
*/
-  private static KafkaSpout createKafkaSpout(String zookeeperUrl, String 
sensorType, SpoutConfig.Offset offset, EnumMap 
kafkaSpoutConfigOptions, SensorParserConfig parserConfig) {
-
+  private static StormKafkaSpout createKafkaSpout(String zkQuorum, String 
sensorType, Optional> kafkaConfigOptional, 
SensorParserConfig parserConfig) {
--- End diff --

yep, we can.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107913632
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
 ---
@@ -106,19 +105,22 @@ public static TopologyBuilder build(String 
zookeeperUrl,
   /**
* Create a spout that consumes tuples from a Kafka topic.
*
-   * @param zookeeperUrlZookeeper URL
+   * @param zkQuorum Zookeeper URL
* @param sensorType  Type of sensor
-   * @param offset  Kafka topic offset where the topology 
will start; BEGINNING, END, WHERE_I_LEFT_OFF
-   * @param kafkaSpoutConfigOptions Configuration options for the kafka 
spout
+   * @param kafkaConfigOptional Configuration options for the kafka 
spout
* @param parserConfigConfiguration for the parser
* @return
*/
-  private static KafkaSpout createKafkaSpout(String zookeeperUrl, String 
sensorType, SpoutConfig.Offset offset, EnumMap 
kafkaSpoutConfigOptions, SensorParserConfig parserConfig) {
-
+  private static StormKafkaSpout createKafkaSpout(String zkQuorum, String 
sensorType, Optional> kafkaConfigOptional, 
SensorParserConfig parserConfig) {
--- End diff --

StormKafkaSpout's return type here will actually be StormKafkaSpout, right?  Can we make that explicit, rather than untyped (and also drop 
the Object.class from the creates assuming the other typing change)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107912514
  
--- Diff: 
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.storm.kafka.flux;
+
+import com.google.common.base.Joiner;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.metron.common.utils.KafkaUtils;
+import org.apache.storm.kafka.spout.*;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.OutputFieldsGetter;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/**
+ * This is a convenience layer on top of the KafkaSpoutConfig.Builder 
available in storm-kafka-client.
+ * The justification for this class is two-fold.  First, there are a lot 
of moving parts and a simplified
+ * approach to constructing spouts is useful.  Secondly, and perhaps more 
importantly, the Builder pattern
+ * is decidedly unfriendly to use inside of Flux.  Finally, we can make 
things a bit more friendly by only requiring
+ * zookeeper and automatically figuring out the brokers for the bootstrap 
server.
+ *
+ * @param  The kafka key type
+ * @param  The kafka value type
+ */
+public class SimpleStormKafkaBuilder extends 
KafkaSpoutConfig.Builder {
+  final static String STREAM = "default";
+
+  /**
+   * The fields exposed by the kafka consumer.  These will show up in the 
Storm tuple.
+   */
+  public enum FieldsConfiguration {
+KEY("key", record -> record.key()),
+VALUE("value", record -> record.value()),
+PARTITION("partition", record -> record.partition()),
+TOPIC("topic", record -> record.topic())
+;
+String fieldName;
+Function recordExtractor;
+
+FieldsConfiguration(String fieldName, Function 
recordExtractor) {
+  this.recordExtractor = recordExtractor;
+  this.fieldName = fieldName;
+}
+
+/**
+ * Return a list of the enums
+ * @param configs
+ * @return
+ */
+public static List toList(String... configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Return a list of the enums from their string representation.
+ * @param configs
+ * @return
+ */
+public static List toList(List configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Construct a Fields object from an iterable of enums.  These fields 
are the fields
+ * exposed in the Storm tuple emitted from the spout.
+ * @param configs
+ * @return
+ */
+public static Fields getFields(Iterable configs) {
+  List fields = new ArrayList<>();
+  for(FieldsConfiguration config : configs) {
+fields.add(config.fieldName);
+  }
+  return new Fields(fields);
+}
+  }
+
+  /**
+   * Build a tuple given the fields and the topic.  We want to use our 
FieldsConfiguration enum
+   * to define what this tuple looks like.
+   * @param  The key type in kafka
+   * @param  The 

[GitHub] incubator-metron issue #490: DO NOT MERGE METRON-797: Pass security.protocol...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/490
  
If you want to review just this work independent of 793 prior to 793 
getting merged, you can find the commits:
* 
[here](https://github.com/apache/incubator-metron/pull/490/commits/5e614efd861a1d059e04ef64914fc185b111e4b6)

Also, I still owe some doc changes around kerberizing vagrant and the CLI 
change to the `start_parser_topology.sh` script.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #490: DO NOT MERGE METRON-797: Pass security.p...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/490

DO NOT MERGE METRON-797: Pass security.protocol and enable auto-renew for 
the storm topologies

## Contributor Comments

METRON-793 migrated the storm topologies to the storm-kafka-client spout, 
which supports kerberos. To complete the kerberos work on the existing 
topologies, we need to be able to enable the spouts and kafka writers to use 
security protocols other than PLAINTEXT. Also, enabling auto-renew plugins for 
storm will enable the topologies to run for extended durations in a kerberized 
cluster.
NOTE: This does not encompass MPack changes to enable kerberos or fix the 
sensors to work with a kerberized kafka. That would be follow-on work.

NOTE: This depends on METRON-793.  As such, this PR includes it, so do not 
merge until 793 is merged.

An acceptance test plan will follow as well as instructions for kerberizing 
vagrant and running the topologies in said environment.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-797

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #490


commit 1a0dd8dafd7e722e9421cebe2f19fe540ccea73d
Author: cstella 
Date:   2017-03-22T14:28:53Z

METRON-793: Migrate to storm-kafka-client kafka spout from storm-kafka

commit 9e2ff8aa7e6113d5aba89ba1c1b5eebb375ffd6d
Author: cstella 
Date:   2017-03-22T14:35:48Z

Updating to remove commented out code.

commit 8a5100b5d9a3abdc464899055de3c0c86e4487d9
Author: cstella 
Date:   2017-03-22T18:06:39Z

Better documentation

commit b6adf40ed7c5ce696d4e9ee7871fcd8bbdca9923
Author: cstella 
Date:   2017-03-22T18:42:23Z

Adding for the spout configuration.

commit 49ccefb5435e6b5057fbba8c5b33f532dbad8904
Author: cstella 

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107904482
  
--- Diff: metron-interface/metron-config/src/app/app.component.ts ---
@@ -0,0 +1,41 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Component} from '@angular/core';
+import './rxjs-operators';
--- End diff --

unused import?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107892970
  
--- Diff: 
metron-interface/metron-config/e2e/sensor-config-readonly/sensor-config-readonly.po.ts
 ---
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { browser, element, by, protractor } from 'protractor/globals';
+import {changeURL, waitForElementVisibility} from '../utils/e2e_util';
+
+export class SensorDetailsPage {
+
+private enableButton;
+private disableButton;
+private startButton;
+private stopButton;
+
+constructor() {
+this.enableButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'ENABLE'));
+this.disableButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'DISABLE'));
+this.startButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'START'));
+this.stopButton = 
element(by.cssContainingText('metron-config-sensor-parser-readonly .btn', 
'STOP'));
+}
+
+clickToggleShowMoreLess(text: string, index: number) {
+return element.all(by.linkText(text)).get(index).click().then(() 
=> {
+browser.sleep(1000);
+return true;
+})
+}
+
+closePane(name: string) {
+return element(by.css('metron-config-sensor-parser-readonly 
.fa-times')).click().then(() => {
+return true;
+});
+}
+
+disableParser() {
+return waitForElementVisibility(this.disableButton).then(() => {
+return this.disableButton.click().then(() => {
+return waitForElementVisibility(this.enableButton).then(() 
=> {
--- End diff --

promises should be really chained rather than nested... see other instances 
in this class too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107905877
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.spec.ts
 ---
@@ -0,0 +1,161 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Inject} from '@angular/core';
+import {async, TestBed, ComponentFixture} from '@angular/core/testing';
+import {Http} from '@angular/http';
+import {GeneralSettingsComponent} from './general-settings.component';
+import {MetronAlerts} from '../shared/metron-alerts';
+import {MetronDialogBox} from '../shared/metron-dialog-box';
+import {GlobalConfigService} from '../service/global-config.service';
+import {GeneralSettingsModule} from './general-settings.module';
+import {Observable} from 'rxjs/Observable';
+import {APP_CONFIG, METRON_REST_CONFIG} from '../app.config';
+import {IAppConfig} from '../app.config.interface';
+
+class MockGlobalConfigService extends GlobalConfigService {
+  _config: any = {};
+  _postSuccess: boolean = true;
+
+  constructor(private http2: Http, @Inject(APP_CONFIG) private config2: 
IAppConfig) {
+super(http2, config2);
+  }
+
+  public post(globalConfig: {}): Observable<{}> {
+if (this._postSuccess) {
+  return Observable.create(observer => {
+observer.next(globalConfig);
+observer.complete();
+  });
+}
+
+return Observable.throw('Error');
+  }
+
+  public get(): Observable<{}> {
+return Observable.create(observer => {
+  observer.next(this._config);
+  observer.complete();
+});
+  }
+}
+
+describe('GeneralSettingsComponent', () => {
+
+  let metronAlerts: MetronAlerts;
+  let metronDialogBox: MetronDialogBox;
+  let component: GeneralSettingsComponent;
+  let globalConfigService: MockGlobalConfigService;
+  let fixture: ComponentFixture;
+  let config = {
+'solr.collection': 'metron',
+'storm.indexingWorkers': 1,
+'storm.indexingExecutors': 2,
+'hdfs.boltBatchSize': 5000,
+'hdfs.boltFieldDelimiter': '|',
+'hdfs.boltFileRotationSize': 5,
+'hdfs.boltCompressionCodecClass': 
'org.apache.hadoop.io.compress.SnappyCodec',
+'hdfs.indexOutput': '/tmp/metron/enriched',
+'kafkaWriter.topic': 'outputTopic',
+'kafkaWriter.keySerializer': 
'org.apache.kafka.common.serialization.StringSerializer',
+'kafkaWriter.valueSerializer': 
'org.apache.kafka.common.serialization.StringSerializer',
+'kafkaWriter.requestRequiredAcks': 1,
+'solrWriter.indexName': 'alfaalfa',
+'solrWriter.shards': 1,
+'solrWriter.replicationFactor': 1,
+'solrWriter.batchSize': 50,
+'fieldValidations': {'field': 'validation'}
+  };
+
+  beforeEach(async(() => {
+
+TestBed.configureTestingModule({
+  imports: [GeneralSettingsModule],
+  providers: [
+{provide: Http},
+MetronAlerts,
+MetronDialogBox,
+{provide: GlobalConfigService, useClass: MockGlobalConfigService},
+{provide: APP_CONFIG, useValue: METRON_REST_CONFIG}
+  ]
+}).compileComponents()
+  .then(() => {
+fixture = TestBed.createComponent(GeneralSettingsComponent);
+component = fixture.componentInstance;
+globalConfigService = 
fixture.debugElement.injector.get(GlobalConfigService);
+metronAlerts = fixture.debugElement.injector.get(MetronAlerts);
+metronDialogBox = 
fixture.debugElement.injector.get(MetronDialogBox);
+  });
+
+  }));
+
+  it('can instantiate GeneralSettingsComponent', async(() => {
+expect(component instanceof GeneralSettingsComponent).toBe(true);
+  }));
+
+  it('should load global config', async(() => {
+globalConfigService._config = config;
+component.ngOnInit();
+
 

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107894517
  
--- Diff: metron-interface/metron-config/e2e/sensor-list/sensor-list.po.ts 
---
@@ -0,0 +1,240 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions andx
+ * limitations under the License.
+ */
+
+import { browser, element, by, protractor } from 'protractor/globals';
+import { waitForElementPresence, waitForStalenessOf } from 
'../utils/e2e_util';
+var Promise = require('bluebird');
+
+export class SensorListPage {
+
+clickOnActionsAndWait(parserNames: string[], clickOnClassName: string, 
waitOnClassName: string) {
+let queueForClick = Promise.resolve();
+parserNames.forEach((name) => {
+queueForClick = queueForClick.then((result) => {
+return this.getIconButton(name, clickOnClassName).click();
+});
+});
+
+return queueForClick.then(() => {
+let promiseArray = [];
+parserNames.map(name => {
+
promiseArray.push(this.waitForElement(this.getIconButton(name, 
waitOnClassName)));
+});
+
+return protractor.promise.all(promiseArray).then(args => {
+return args;
+});
+});
+}
+
+clickOnDropdownAndWait(parserNames: string[], dropDownLinkName: 
string, waitOnClassName: string) {
+return protractor.promise.all([this.toggleSelectAll(), 
this.toggleDropdown()]).then(() => {
+
+return element(by.css('span[data-action=\"'+ dropDownLinkName 
+'\"]')).click().then(() => {
+let promiseArray = [];
+parserNames.map(name => {
+
promiseArray.push(this.waitForElement(this.getIconButton(name, 
waitOnClassName)));
--- End diff --

more efficient to avoid side-effects here, since map returns the result of 
a transformation, e.g. promiseArray = parserNames.map( (name) => x ) 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107907080
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.ts
 ---
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { Component, OnInit, ViewChild } from '@angular/core';
+import {GlobalConfigService} from '../service/global-config.service';
+import {MetronAlerts} from '../shared/metron-alerts';
+import {MetronDialogBox} from '../shared/metron-dialog-box';
+
+@Component({
+  selector: 'metron-config-general-settings',
+  templateUrl: './general-settings.component.html',
+  styleUrls: ['./general-settings.component.scss']
+})
+export class GeneralSettingsComponent implements OnInit {
+
+  @ViewChild('globalSettingsForm') globalSettingsForm: any;
+
+  globalConfig: {} = {
+'es.date.format': '-'
+  };
+
+  private fieldValidations: string;
+
+  constructor(private globalConfigService: GlobalConfigService, private 
metronAlerts: MetronAlerts,
--- End diff --

Not sure why this class is serialising to JSON for fieldValidations. Feels 
like unnecessary overhead.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107719953
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.html
 ---
@@ -0,0 +1,106 @@
+
+
+
+
+  
+  
+{{selectedSensorName}}
+  
+
+  
+
+  
+
+
+  
+  
+
+
+
+
+
+  
+{{ item.value }}
+  
+
+  
+{{ item.label 
}}
+{{ 
sensorParserConfigHistory[item.value] ? sensorParserConfigHistory[item.value] : 
"-" }}
+{{ kafkaTopic[item.value] ? kafkaTopic[item.value] : "-" 
}}
+{{ topologyStatus[item.value] ? 
topologyStatus[item.value] : "-" }}
+
+  Grok Statement
+  
+  
+  show more
+
+
+
+
+  
+Transforms
+
+  
+{{ results }}
+{{ 
transformsConfigMap[results] }}
+  
+
+  
+
+  
+Transforms
+
+  
+  show more
+
+
+  AGGREGATOR{{sensorEnrichmentConfig.threatIntel.triageConfig.aggregator}}
+  
+
+  NAMESCORE
+  
+{{ 
getDisplayName(riskLevelRule) }}
+{{ 
riskLevelRule.score }}
+  
+
+  
+  {{ 
getRuleDisplayName() }}
+  show more
--- End diff --

should this be encapsulated in a controller function?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107894592
  
--- Diff: metron-interface/metron-config/e2e/sensor-list/sensor-list.po.ts 
---
@@ -0,0 +1,240 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions andx
+ * limitations under the License.
+ */
+
+import { browser, element, by, protractor } from 'protractor/globals';
+import { waitForElementPresence, waitForStalenessOf } from 
'../utils/e2e_util';
+var Promise = require('bluebird');
+
+export class SensorListPage {
+
+clickOnActionsAndWait(parserNames: string[], clickOnClassName: string, 
waitOnClassName: string) {
+let queueForClick = Promise.resolve();
+parserNames.forEach((name) => {
+queueForClick = queueForClick.then((result) => {
+return this.getIconButton(name, clickOnClassName).click();
+});
+});
+
+return queueForClick.then(() => {
+let promiseArray = [];
+parserNames.map(name => {
+
promiseArray.push(this.waitForElement(this.getIconButton(name, 
waitOnClassName)));
+});
+
+return protractor.promise.all(promiseArray).then(args => {
+return args;
+});
+});
+}
+
+clickOnDropdownAndWait(parserNames: string[], dropDownLinkName: 
string, waitOnClassName: string) {
+return protractor.promise.all([this.toggleSelectAll(), 
this.toggleDropdown()]).then(() => {
+
+return element(by.css('span[data-action=\"'+ dropDownLinkName 
+'\"]')).click().then(() => {
+let promiseArray = [];
+parserNames.map(name => {
+
promiseArray.push(this.waitForElement(this.getIconButton(name, 
waitOnClassName)));
+});
+
+return protractor.promise.all(promiseArray).then(args => {
+return this.toggleSelectAll().then(() => {
+return args;
+})
+});
+});
+});
+}
+
+closePane(className: string ='.close-button') {
+return this.waitForElement(element(by.css(className))).then(() => {
+return element(by.css(className)).click();
+});
+}
+
+disableParsers(names: string[]) {
+return this.clickOnActionsAndWait(names, 'i.fa-ban', 
'i.fa-check-circle-o');
+}
+
+disableParsersFromDropdown(names: string[]) {
+return this.clickOnDropdownAndWait(names, 'Disable', 
'i.fa-check-circle-o');
+}
+
+deleteParser(name: string) {
+return this.getIconButton(name, '.fa-trash-o').click().then(() => {
+browser.sleep(1000);
+return element(by.css('.metron-dialog 
.btn-primary')).click().then(() => {
--- End diff --

promise chaining 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107902092
  
--- Diff: metron-interface/metron-config/scripts/start_management_ui.sh ---
@@ -0,0 +1,27 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+SCRIPTS_ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+export METRON_REST_URL=${METRON_REST_URL:-http://localhost:8080}
+export MANAGEMENT_UI_PORT=${MANAGEMENT_UI_PORT:-4200}
+npm version > /dev/null
+if [ $? -eq 0 ]; then
+npm install http-server
--- End diff --

should this be in packages.json instead to avoid the need for npm on 
production nodes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107719782
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.html
 ---
@@ -0,0 +1,106 @@
+
+
+
+
+  
+  
+{{selectedSensorName}}
+  
+
+  
+
+  
+
+
+  
+  
+
+
+
+
+
+  
+{{ item.value }}
+  
+
+  
+{{ item.label 
}}
+{{ 
sensorParserConfigHistory[item.value] ? sensorParserConfigHistory[item.value] : 
"-" }}
+{{ kafkaTopic[item.value] ? kafkaTopic[item.value] : "-" 
}}
+{{ topologyStatus[item.value] ? 
topologyStatus[item.value] : "-" }}
+
+  Grok Statement
+  
+  
+  show more
+
+
+
+
+  
+Transforms
+
+  
+{{ results }}
+{{ 
transformsConfigMap[results] }}
+  
+
+  
+
+  
+Transforms
+
+  
+  show more
+
+
+  AGGREGATOR{{sensorEnrichmentConfig.threatIntel.triageConfig.aggregator}}
+  
+
+  NAMESCORE
+  
+{{ 
getDisplayName(riskLevelRule) }}
+{{ 
riskLevelRule.score }}
+  
+
+  
+  {{ 
getRuleDisplayName() }}
+  show more
+
+  
+
+  
+
+
+  
+  
+EDIT
+
+START
+STOP
--- End diff --

controller logic?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107904759
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.html
 ---
@@ -0,0 +1,107 @@
+
+
+  
+
+   General Settings 
+
+
+
+  
+
+  
+  Elasticsearch  
+
+  INDEX DATE FORMAT
+  
+
+
+  
+
+  
+  Validation  
+
+
+  FIELD
+  
+
+
+  
+
+
+
+

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107904887
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.html
 ---
@@ -0,0 +1,107 @@
+
+
+  
+
+   General Settings 
+
+
+
+  
+
+  
+  Elasticsearch  
+
+  INDEX DATE FORMAT
+  
+
+
+  
+
+  
+  Validation  
+
+
+  FIELD
+  
+
+
+  
+
+
+
+
+  
+
+
+  
+
+  
--- End diff --

better handled with styling css padding rather than linebreaks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107900720
  
--- Diff: metron-interface/metron-config/package.json ---
@@ -0,0 +1,65 @@
+{
+  "name": "clitest",
+  "version": "0.0.0",
--- End diff --

Should track metron version


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107905795
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.spec.ts
 ---
@@ -0,0 +1,161 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Inject} from '@angular/core';
+import {async, TestBed, ComponentFixture} from '@angular/core/testing';
+import {Http} from '@angular/http';
+import {GeneralSettingsComponent} from './general-settings.component';
+import {MetronAlerts} from '../shared/metron-alerts';
+import {MetronDialogBox} from '../shared/metron-dialog-box';
+import {GlobalConfigService} from '../service/global-config.service';
+import {GeneralSettingsModule} from './general-settings.module';
+import {Observable} from 'rxjs/Observable';
+import {APP_CONFIG, METRON_REST_CONFIG} from '../app.config';
+import {IAppConfig} from '../app.config.interface';
+
+class MockGlobalConfigService extends GlobalConfigService {
+  _config: any = {};
+  _postSuccess: boolean = true;
+
+  constructor(private http2: Http, @Inject(APP_CONFIG) private config2: 
IAppConfig) {
+super(http2, config2);
+  }
+
+  public post(globalConfig: {}): Observable<{}> {
+if (this._postSuccess) {
+  return Observable.create(observer => {
+observer.next(globalConfig);
+observer.complete();
+  });
+}
+
+return Observable.throw('Error');
+  }
+
+  public get(): Observable<{}> {
+return Observable.create(observer => {
+  observer.next(this._config);
+  observer.complete();
+});
+  }
+}
+
+describe('GeneralSettingsComponent', () => {
+
+  let metronAlerts: MetronAlerts;
+  let metronDialogBox: MetronDialogBox;
+  let component: GeneralSettingsComponent;
+  let globalConfigService: MockGlobalConfigService;
+  let fixture: ComponentFixture;
+  let config = {
+'solr.collection': 'metron',
+'storm.indexingWorkers': 1,
+'storm.indexingExecutors': 2,
+'hdfs.boltBatchSize': 5000,
+'hdfs.boltFieldDelimiter': '|',
+'hdfs.boltFileRotationSize': 5,
+'hdfs.boltCompressionCodecClass': 
'org.apache.hadoop.io.compress.SnappyCodec',
+'hdfs.indexOutput': '/tmp/metron/enriched',
+'kafkaWriter.topic': 'outputTopic',
+'kafkaWriter.keySerializer': 
'org.apache.kafka.common.serialization.StringSerializer',
+'kafkaWriter.valueSerializer': 
'org.apache.kafka.common.serialization.StringSerializer',
+'kafkaWriter.requestRequiredAcks': 1,
+'solrWriter.indexName': 'alfaalfa',
+'solrWriter.shards': 1,
+'solrWriter.replicationFactor': 1,
+'solrWriter.batchSize': 50,
+'fieldValidations': {'field': 'validation'}
+  };
+
+  beforeEach(async(() => {
+
+TestBed.configureTestingModule({
+  imports: [GeneralSettingsModule],
+  providers: [
+{provide: Http},
+MetronAlerts,
+MetronDialogBox,
+{provide: GlobalConfigService, useClass: MockGlobalConfigService},
+{provide: APP_CONFIG, useValue: METRON_REST_CONFIG}
+  ]
+}).compileComponents()
+  .then(() => {
+fixture = TestBed.createComponent(GeneralSettingsComponent);
+component = fixture.componentInstance;
+globalConfigService = 
fixture.debugElement.injector.get(GlobalConfigService);
+metronAlerts = fixture.debugElement.injector.get(MetronAlerts);
+metronDialogBox = 
fixture.debugElement.injector.get(MetronDialogBox);
+  });
+
+  }));
+
+  it('can instantiate GeneralSettingsComponent', async(() => {
+expect(component instanceof GeneralSettingsComponent).toBe(true);
+  }));
+
+  it('should load global config', async(() => {
+globalConfigService._config = config;
+component.ngOnInit();
+
 

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107904541
  
--- Diff: metron-interface/metron-config/src/app/app.component.ts ---
@@ -0,0 +1,41 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Component} from '@angular/core';
+import './rxjs-operators';
+import {AuthenticationService} from './service/authentication.service';
+import 'jquery';
--- End diff --

unused import? and $ declaration? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107906446
  
--- Diff: 
metron-interface/metron-config/src/app/general-settings/general-settings.component.ts
 ---
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { Component, OnInit, ViewChild } from '@angular/core';
+import {GlobalConfigService} from '../service/global-config.service';
+import {MetronAlerts} from '../shared/metron-alerts';
+import {MetronDialogBox} from '../shared/metron-dialog-box';
+
+@Component({
+  selector: 'metron-config-general-settings',
+  templateUrl: './general-settings.component.html',
+  styleUrls: ['./general-settings.component.scss']
+})
+export class GeneralSettingsComponent implements OnInit {
+
+  @ViewChild('globalSettingsForm') globalSettingsForm: any;
--- End diff --

Type should be limited to the child component type


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107893194
  
--- Diff: 
metron-interface/metron-config/e2e/sensor-config/sensor-config.po.ts ---
@@ -0,0 +1,243 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { browser, element, by, protractor } from 'protractor/globals';
+import {changeURL, waitForElementVisibility, waitForElementInVisibility, 
waitForElementPresence} from '../utils/e2e_util';
+
+export class SensorConfigPage {
+
+  _getEnrichmentsMultipleInput() {
--- End diff --

_ usually indicated a private function, should this also be declared 
private?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107892229
  
--- Diff: metron-interface/metron-config/angular-cli.json ---
@@ -0,0 +1,51 @@
+{
+  "project": {
+"version": "1.0.0-beta.15",
--- End diff --

Should we version this to match Metron versions? It's likely to run lock 
step with Metron from a release perspective.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/484#discussion_r107719261
  
--- Diff: 
metron-interface/metron-config/src/app/sensors/sensor-grok/sensor-grok.component.html
 ---
@@ -0,0 +1,42 @@
+
+
+
+Grok Validator
+
+
+
+
+STATEMENT
+ 

+
+
+TEST
--- End diff --

should this logic be in controller?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #489: METRON-623: Management UI

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/incubator-metron/pull/489

METRON-623: Management UI

## Contributor Comments
This PR includes the latest version of the Management UI.  Instructions for 
building and installing are located in 
metron-interface/metron-config/README.md.  

Here are the commands I used to deploy to Quick Dev:

1. Install REST application:
```
scp ./metron-interface/metron-rest/target/metron-rest-0.3.1-archive.tar.gz 
root@node1:/usr/metron/0.3.1/
ssh root@node1
cd /usr/metron/0.3.1/
tar -xf metron-rest-0.3.1-archive.tar.gz
java -jar ./lib/metron-rest-0.3.1.jar --spring.profiles.active=vagrant,dev 
--server.port=8082
```
2. Install the Management UI:
```
scp 
./metron-interface/metron-config/target/metron-config-0.3.1-archive.tar.gz 
root@node1:/usr/metron/0.3.1/
ssh root@node1
cd /usr/metron/0.3.1/
tar -xf metron-config-0.3.1-archive.tar.gz
yum install -y npm
npm install npm@3.8.9 -g
export METRON_REST_URL=http://node1:8082
./bin/start_management_ui.sh
```
The UI should be available at http://node1:4200 (login with user/password).

So far this UI has only been dev tested and there will likely be some bugs 
found.  If reviewers feel they have found a critical bug that should be fixed 
before this can be accepted, please state that and we will address it ASAP.  
Otherwise I will create Jiras for bugs/issues that can be done as follow on 
work.

This is a very large pull request that contains mostly client-side code.  
The application was built on the [Angular 
2](https://angular.io/docs/ts/latest/) framework using [Angular 
CLI](https://github.com/angular/angular-cli).

Changes that are outside of metron-interface/metron-config (Management UI 
module) include:

- Addition of "/api/v1/grok/get/statement" REST endpoint that gets a grok 
statement from the classpath instead of HDFS
- A couple fields added to the Storm topology status response (emitted and 
acked tuples)
- Updated REST documentation
- Apache Rat exclusions added to the top level pom.xml

There are a couple tasks still outstanding so don't merge until those are 
complete.  These tasks should not affect functional testing or code review.  
They include:

- add architectural documentation to the README

I will keep the thread updated as these are completed.  Thanks for 
reviewing!

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please 

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[merrimanr]

Github user merrimanr closed the pull request at:

https://github.com/apache/incubator-metron/pull/484


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[merrimanr]

GitHub user merrimanr reopened a pull request:

https://github.com/apache/incubator-metron/pull/484

METRON-623: Management UI

## Contributor Comments
This PR includes the latest version of the Management UI.  Instructions for 
building and installing are located in 
metron-interface/metron-config/README.md.  

Here are the commands I used to deploy to Quick Dev:

1. Install REST application:
```
scp ./metron-interface/metron-rest/target/metron-rest-0.3.1-archive.tar.gz 
root@node1:/usr/metron/0.3.1/
ssh root@node1
cd /usr/metron/0.3.1/
tar -xf metron-rest-0.3.1-archive.tar.gz
java -jar ./lib/metron-rest-0.3.1.jar --spring.profiles.active=vagrant,dev 
--server.port=8082
```
2. Install the Management UI:
```
scp 
./metron-interface/metron-config/target/metron-config-0.3.1-archive.tar.gz 
root@node1:/usr/metron/0.3.1/
ssh root@node1
cd /usr/metron/0.3.1/
tar -xf metron-config-0.3.1-archive.tar.gz
yum install -y npm
npm install npm@3.8.9 -g
export METRON_REST_URL=http://node1:8082
./bin/start_management_ui.sh
```
The UI should be available at http://node1:4200 (login with user/password).

So far this UI has only been dev tested and there will likely be some bugs 
found.  If reviewers feel they have found a critical bug that should be fixed 
before this can be accepted, please state that and we will address it ASAP.  
Otherwise I will create Jiras for bugs/issues that can be done as follow on 
work.

This is a very large pull request that contains mostly client-side code.  
The application was built on the [Angular 
2](https://angular.io/docs/ts/latest/) framework using [Angular 
CLI](https://github.com/angular/angular-cli).

Changes that are outside of metron-interface/metron-config (Management UI 
module) include:

- Addition of "/api/v1/grok/get/statement" REST endpoint that gets a grok 
statement from the classpath instead of HDFS
- A couple fields added to the Storm topology status response (emitted and 
acked tuples)
- Updated REST documentation
- Apache Rat exclusions added to the top level pom.xml

There are a couple tasks still outstanding so don't merge until those are 
complete.  These tasks should not affect functional testing or code review.  
They include:

- add angular unit tests to the Maven build cycle (tests pass and can be 
run with `npm test` in the metron-config directory, they just don't get 
executed on a top level build yet)
- optimize build process and dependencies (our build process is already 
painful, need to ensure this doesn't make it worse)
- add architectural documentation to the README
- resolve error when building the site-book

I will keep the thread updated as these are completed.  Thanks for 
reviewing!

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related 

[GitHub] incubator-metron pull request #484: METRON-623: Management UI

[merrimanr]

Github user merrimanr closed the pull request at:

https://github.com/apache/incubator-metron/pull/484


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #488: METRON-796: Mpack uses wrong group for owning H...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/488
  
I agree the topologies should run as the metron user, but this is just to 
get things back in a working state again (and it already used to be this way, 
so this isn't opening things up more than it was a couple weeks ago).

I actually thought there was a separate Jira for running as the Metron 
user, but the one I was thinking of is 
https://issues.apache.org/jira/browse/METRON-349.  The ticket should really be 
to consolidate everything under the metron user with appropriate ownership.  I 
don't have a preference for updating that ticket or closing it as a new one 
(and I'm not sure which the community would prefer).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #488: METRON-796: Mpack uses wrong group for owning H...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/incubator-metron/pull/488
  
Opening this to the hadoop group feels wrong from a security perspective. 
That gives all the other hadoop users too much write access to the data store. 
A better solution would be to have the topologies run as a metron user rather 
than as storm, or to add the storm user to the metron group to maintain a least 
access required principal.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #488: METRON-796: Mpack uses wrong group for o...

[justinleet]

GitHub user justinleet opened a pull request:

https://github.com/apache/incubator-metron/pull/488

METRON-796: Mpack uses wrong group for owning HDFS directories

## Contributor Comments
Reverts the group owner of a couple HDFS directories to be the hadoop 
group, rather than the metron group (which is just metron).  Right now, the 
topologies run as the storm user (which belongs to the hadoop group), and 
therefore didn't have permission to write to HDFS (including in quick and full 
dev).  This sets HDFS ownership to metron:hadoop, which lets it be handled 
appropriately.

Other items, such as configs and installation files, were just left as the 
metron group.

To test, just run up a dev environment and ensure files are being written 
and ownership makes sense (/apps/metron/indexing/indexed is metron:hadoop with 
755 perms).  The individual sensors will be owned by storm:hadoop (proving that 
writes work).

For example:
```
[vagrant@node1 ~]$ hdfs dfs -ls /apps/metron/indexing
Found 1 items
drwxrwxr-x   - metron hadoop  0 2017-03-24 12:57 
/apps/metron/indexing/indexed
[vagrant@node1 ~]$ hdfs dfs -ls /apps/metron/indexing/indexed
Found 3 items
drwxrwxr-x   - storm hadoop  0 2017-03-24 13:01 
/apps/metron/indexing/indexed/bro
drwxrwxr-x   - storm hadoop  0 2017-03-24 13:01 
/apps/metron/indexing/indexed/error
drwxrwxr-x   - storm hadoop  0 2017-03-24 13:01 
/apps/metron/indexing/indexed/snort
[vagrant@node1 ~]$ hdfs dfs -ls /apps/metron/indexing/indexed/bro
Found 1 items
-rw-r--r--   1 storm hadoop 211393 2017-03-24 13:01 
/apps/metron/indexing/indexed/bro/enrichment-null-0-0-1490360489968.json
```

As a note, metron_group existed twice in params_linux.py, so only the first 
instance is changed to hadoop_group and pulled appropriately.  The second is 
left as-is.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/incubator-metron METRON-796

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #488


commit ec7d070524334603e1712b4649e5600ded450284
Author: justinjleet 
Date:   2017-03-24T00:55:23Z

Updating group perms




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---