[GitHub] metron issue #526: Metron-846: Add E2E tests for metron management ui

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/526
  
I am testing and updating this PR to get it to work on latest master. 


---

[jira] [Commented] (METRON-1563) Initial Feature Branch Work

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474822#comment-16474822
 ] 

ASF GitHub Bot commented on METRON-1563:


GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/1014

METRON-1563 : Base Stellar assign for feature branch

repackage: 
https://github.com/apache/metron/pull/687

Please sanity check and see that PR

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron stellar_assign

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1014.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1014


commit 37587689e9feadeb54465aa7319f114a4e11255b
Author: cstella 
Date:   2017-12-12T20:24:07Z

METRON-1306: When index template install fails, we should fail the install 
closes apache/incubator-metron#834

commit 6ca08b9f598b649d60c6b6468164a374b7f6f555
Author: MohanDV 
Date:   2017-12-13T15:55:04Z

METRON-1343 Swagger UI for User Controller needs request method (MohanDV 
via ottobackwards) closes apache/metron#862

commit d9ed1bad1f5b2e2471fbea11353f2947f7f52e13
Author: nickwallen 
Date:   2017-12-14T15:59:27Z

METRON-1349 Full Dev Builds Metron Twice (nickwallen) closes 
apache/metron#866

commit c4cee6af64eda4db4da3eff86abab7d4ae4ec56a
Author: mmiklavc 
Date:   2017-12-14T20:29:49Z

METRON-1345: Update EC2 README for custom Ansible (mmiklavc via mmiklavc) 
closes apache/metron#859

commit d446da8e707b1069576b049452484e088a3eeede
Author: nickwallen 
Date:   2017-12-19T17:42:39Z

METRON-1372 Validate JIRA for Releases (nickwallen) closes apache/metron#874

commit adb024070d5d098909fb3800875d0042aeb27c92
Author: ottobackwards 
Date:   2017-12-19T20:13:22Z

METRON-1374 Script the RC checking process (ottobackwards) closes 
apache/metron#876

commit 3f0b1b7b4a002d3f364bd2aee7b5921c0435c4a4
Author: cstella 
Date:   2017-12-20T14:30:03Z

METRON-1350: Add reservoir sampling functions to Stellar closes 
apache/incubator-metron#867

commit 196da12c43337d019a52b99bf6178fbda45f886d
Author: nickwallen 
Date:   2017-12-21T14:04:49Z

METRON-1348 Metron Service Checks Use Wrong Hostname (nickwallen) closes 
apache/metron#864

commit 76bed5d754fcf358809f0be7a034758b9b20fc5e
Author: cstella 
Date:   2017-12-21T21:49:31Z

METRON-1365: Allow PROFILE_GET to return a default value for a profile and 
entity that does not have a value written. closes apache/incubator-metron#871

commit 3612a89216bd57c40a1bc3e27853c6146b471e1e
Author: ottobackwards 
Date:   2017-12-25T20:44:45Z

METRON-1376 RC Check Script should have named parameters (ottobackwards via 
nickwallen) closes apache/metron#877

commit fc8723e461d655e315d0b51acd1a31f82b4efd1f
Author: nickwallen 
Date:   2017-12-27T18:25:53Z

METRON-1351 Create Installable Packages for Ubuntu Trusty (nickwallen) 
closes apache/metron#868

commit 0518408513ed54df8dbe234027b353bed2e61943
Author: mattf-horton 
Date:   2017-12-31T22:01:29Z

METRON-1373 RAT failure for metron-interface/metron-alerts (mattf-horton) 
closes apache/metron#875

commit 3b10f84cc49993a1c5917f54be6ca313c8d780c4
Author: justinleet 
Date:   2018-01-01T23:49:57Z

METRON-1071 Create CONTRIBUTING.md (justinleet) closes apache/metron#881

commit 2d9d7a5f6302267edafba772268145e76751795c
Author: justinleet 
Date:   2018-01-02T16:38:54Z

METRON-1381 Add Apache license to MD files and remove the Rat exclusion 
(justinleet) closes apache/metron#883

commit 8a61b96b6b01b247a8ff8d800730378b8da23471
Author: mattf-horton 
Date:   2018-01-02T17:55:52Z

METRON-1384 Increment master version number to 0.4.3 for on-going 
development (mattf-horton via nickwallen) closes apache/metron#885

commit 01c26a77b1041204b0bbbc544cc0a5d02e9339a8
Author: nickwallen 
Date:   2018-01-03T14:52:57Z

METRON-1362 Improve Metron Deployment README (nickwallen) closes 
apache/metron#869

commit 3381b853dca1c08a7a083593045dec2c7d4d92db
Author: mattf-horton 
Date:   2018-01-04T20:30:30Z

METRON-1388 update public web site to point at 0.4.2 new release 
(mattf-horton) closes apache/metron#887

commit 9108072756b6ffeedade985d3cd52ef7338cd61a
Author: merrimanr 
Date:   2018-01-08T14:18:30Z

METRON-1385 Missing properties in index template causes 
ElasticsearchColumnMetadataDao.getColumnMetadata to fail (merrimanr) closes 
apache/metron#886

commit 0996b7348eca14fea1b1b3c4dd57861b3a30bdeb
Author: cstella 
Date:   

[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1014
  
@cestella I rebased this on the new feature branch ( after rebasing on the 
same master ) and I get all of these other commits.  I don't know how to get 
rid of them?


---

[GitHub] metron pull request #1014: METRON-1563 : Base Stellar assign for feature bra...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/1014

METRON-1563 : Base Stellar assign for feature branch

repackage: 
https://github.com/apache/metron/pull/687

Please sanity check and see that PR

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron stellar_assign

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1014.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1014


commit 37587689e9feadeb54465aa7319f114a4e11255b
Author: cstella 
Date:   2017-12-12T20:24:07Z

METRON-1306: When index template install fails, we should fail the install 
closes apache/incubator-metron#834

commit 6ca08b9f598b649d60c6b6468164a374b7f6f555
Author: MohanDV 
Date:   2017-12-13T15:55:04Z

METRON-1343 Swagger UI for User Controller needs request method (MohanDV 
via ottobackwards) closes apache/metron#862

commit d9ed1bad1f5b2e2471fbea11353f2947f7f52e13
Author: nickwallen 
Date:   2017-12-14T15:59:27Z

METRON-1349 Full Dev Builds Metron Twice (nickwallen) closes 
apache/metron#866

commit c4cee6af64eda4db4da3eff86abab7d4ae4ec56a
Author: mmiklavc 
Date:   2017-12-14T20:29:49Z

METRON-1345: Update EC2 README for custom Ansible (mmiklavc via mmiklavc) 
closes apache/metron#859

commit d446da8e707b1069576b049452484e088a3eeede
Author: nickwallen 
Date:   2017-12-19T17:42:39Z

METRON-1372 Validate JIRA for Releases (nickwallen) closes apache/metron#874

commit adb024070d5d098909fb3800875d0042aeb27c92
Author: ottobackwards 
Date:   2017-12-19T20:13:22Z

METRON-1374 Script the RC checking process (ottobackwards) closes 
apache/metron#876

commit 3f0b1b7b4a002d3f364bd2aee7b5921c0435c4a4
Author: cstella 
Date:   2017-12-20T14:30:03Z

METRON-1350: Add reservoir sampling functions to Stellar closes 
apache/incubator-metron#867

commit 196da12c43337d019a52b99bf6178fbda45f886d
Author: nickwallen 
Date:   2017-12-21T14:04:49Z

METRON-1348 Metron Service Checks Use Wrong Hostname (nickwallen) closes 
apache/metron#864

commit 76bed5d754fcf358809f0be7a034758b9b20fc5e
Author: cstella 
Date:   2017-12-21T21:49:31Z

METRON-1365: Allow PROFILE_GET to return a default value for a profile and 
entity that does not have a value written. closes apache/incubator-metron#871

commit 3612a89216bd57c40a1bc3e27853c6146b471e1e
Author: ottobackwards 
Date:   2017-12-25T20:44:45Z

METRON-1376 RC Check Script should have named parameters (ottobackwards via 
nickwallen) closes apache/metron#877

commit fc8723e461d655e315d0b51acd1a31f82b4efd1f
Author: nickwallen 
Date:   2017-12-27T18:25:53Z

METRON-1351 Create Installable Packages for Ubuntu Trusty (nickwallen) 
closes apache/metron#868

commit 0518408513ed54df8dbe234027b353bed2e61943
Author: mattf-horton 
Date:   2017-12-31T22:01:29Z

METRON-1373 RAT failure for metron-interface/metron-alerts (mattf-horton) 
closes apache/metron#875

commit 3b10f84cc49993a1c5917f54be6ca313c8d780c4
Author: justinleet 
Date:   2018-01-01T23:49:57Z

METRON-1071 Create CONTRIBUTING.md (justinleet) closes apache/metron#881

commit 2d9d7a5f6302267edafba772268145e76751795c
Author: justinleet 
Date:   2018-01-02T16:38:54Z

METRON-1381 Add Apache license to MD files and remove the Rat exclusion 
(justinleet) closes apache/metron#883

commit 8a61b96b6b01b247a8ff8d800730378b8da23471
Author: mattf-horton 
Date:   2018-01-02T17:55:52Z

METRON-1384 Increment master version number to 0.4.3 for on-going 
development (mattf-horton via nickwallen) closes apache/metron#885

commit 01c26a77b1041204b0bbbc544cc0a5d02e9339a8
Author: nickwallen 
Date:   2018-01-03T14:52:57Z

METRON-1362 Improve Metron Deployment README (nickwallen) closes 
apache/metron#869

commit 3381b853dca1c08a7a083593045dec2c7d4d92db
Author: mattf-horton 
Date:   2018-01-04T20:30:30Z

METRON-1388 update public web site to point at 0.4.2 new release 
(mattf-horton) closes apache/metron#887

commit 9108072756b6ffeedade985d3cd52ef7338cd61a
Author: merrimanr 
Date:   2018-01-08T14:18:30Z

METRON-1385 Missing properties in index template causes 
ElasticsearchColumnMetadataDao.getColumnMetadata to fail (merrimanr) closes 
apache/metron#886

commit 0996b7348eca14fea1b1b3c4dd57861b3a30bdeb
Author: cstella 
Date:   2018-01-08T14:30:58Z

METRON-1377: Stellar function to generate typosquatted domains (similar to 
dnstwist) closes apache/incubator-metron#878

commit e821391895f0f3523b4e05ae5d639c13b0113855
Author: mmiklavc 
Date:   

[GitHub] metron pull request #687: METRON-1563 [DISCUSS] Add Assignment to Stellar La...

[ottobackwards]

Github user ottobackwards closed the pull request at:

https://github.com/apache/metron/pull/687


---

[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474796#comment-16474796
 ] 

ASF GitHub Bot commented on METRON-1551:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1012


> Profiler Should Not Use Java Serialization
> --
>
> Key: METRON-1551
> URL: https://issues.apache.org/jira/browse/METRON-1551
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> When running the Profiler in a topology where serialization occurs, the 
> following error happens.  This can occur when the number of workers is 
> greater than 1.
> The topology should not be using Java serialization for serializing tuple 
> values as this will negatively impact performance. 
> {code}
> 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] 
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  ... 6 more
> Caused by: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
> ~[?:1.8.0_162]
>  at 
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
> ~[kryo-3.0.3.jar:?]
>  at 
> 

[GitHub] metron pull request #1012: METRON-1551 Profiler Should Not Use Java Serializ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1012


---

[jira] [Commented] (METRON-1552) Add gzip file validation check to the geo loader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474783#comment-16474783
 ] 

ASF GitHub Bot commented on METRON-1552:


Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r188093107
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
+System.exit(4);
--- End diff --

The existing exit call seemed reasonable to me if only because I'd expect 
this to be user error, not due to issues with an external system.


> Add gzip file validation check to the geo loader
> 
>
> Key: METRON-1552
> URL: https://issues.apache.org/jira/browse/METRON-1552
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1011: METRON-1552: Add gzip file validation check to th...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/1011#discussion_r188093107
  
--- Diff: 
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
 ---
@@ -167,26 +184,38 @@ protected void loadGeoIpDatabase(CommandLine cli) 
throws IOException {
 System.out.println("Successfully created and updated new GeoIP 
information");
   }
 
-  protected File downloadGeoFile(String urlStr, String tmpDir) {
+  protected File downloadGeoFile(String urlStr, String tmpDir, int  
numRetries) {
 File localFile = null;
-try {
-  URL url = new URL(urlStr);
-  localFile = new File(tmpDir + new File(url.getPath()).getName());
+int attempts = 0;
+boolean valid = false;
+while (attempts <= numRetries) {
+  try {
+URL url = new URL(urlStr);
+localFile = new File(tmpDir + new File(url.getPath()).getName());
 
-  System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
-  if (localFile.exists() && !localFile.delete()) {
-System.err.println("File already exists locally and can't be 
deleted.  Please delete before continuing");
-System.exit(3);
+System.out.println("Downloading " + url.toString() + " to " + 
localFile.getAbsolutePath());
+if (localFile.exists() && !localFile.delete()) {
+  System.err.println(
+  "File already exists locally and can't be deleted.  Please 
delete before continuing");
+  System.exit(3);
+}
+FileUtils.copyURLToFile(url, localFile, 5000, 1);
+if (!CompressionStrategies.GZIP.test(localFile)) {
+  throw new IOException("Invalid Gzip file");
+}
+  } catch (MalformedURLException e) {
+System.err.println("Malformed URL - aborting: " + e);
+e.printStackTrace();
+System.exit(4);
--- End diff --

The existing exit call seemed reasonable to me if only because I'd expect 
this to be user error, not due to issues with an external system.


---

[jira] [Commented] (METRON-1548) Alerts UI: Remove hardcoded source:type and other fields that may vary

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474753#comment-16474753
 ] 

ASF GitHub Bot commented on METRON-1548:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/1010#discussion_r188086075
  
--- Diff: 
metron-interface/metron-alerts/src/app/service/global-config.service.ts ---
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Injectable, Inject} from '@angular/core';
+import {Http, Headers, RequestOptions, Response, ResponseOptions} from 
'@angular/http';
+import {Observable} from 'rxjs/Observable';
+import {HttpUtil} from '../utils/httpUtil';
+
+@Injectable()
+export class GlobalConfigService {
+  url = 'api/v1/global/config';
+  defaultHeaders = {'Content-Type': 'application/json', 
'X-Requested-With': 'XMLHttpRequest'};
+
+  private globalConfig = {};
+
+  constructor(private http: Http) {
--- End diff --

I think this constructor was carried over from metron-config and the 
property assignments are not needed.  I think it should just be empty.


> Alerts UI: Remove hardcoded source:type and other fields that may vary
> --
>
> Key: METRON-1548
> URL: https://issues.apache.org/jira/browse/METRON-1548
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Priority: Major
>
> In Solr, we use source.type instead of source:type (which was originally due 
> to ES limitations, we'd prefer it to be common across both). However, it's 
> hardcoded in the alerts UI to be source:type, so nothing in the UI has a 
> source type and it breaks other things like metaalerts (which rely on source 
> type for being able to pull the data together).
> Any other configs / queries that would cause similar problems should be 
> updated appropriately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1010: METRON-1548: Remove hardcoded source:type from Al...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/1010#discussion_r188086075
  
--- Diff: 
metron-interface/metron-alerts/src/app/service/global-config.service.ts ---
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Injectable, Inject} from '@angular/core';
+import {Http, Headers, RequestOptions, Response, ResponseOptions} from 
'@angular/http';
+import {Observable} from 'rxjs/Observable';
+import {HttpUtil} from '../utils/httpUtil';
+
+@Injectable()
+export class GlobalConfigService {
+  url = 'api/v1/global/config';
+  defaultHeaders = {'Content-Type': 'application/json', 
'X-Requested-With': 'XMLHttpRequest'};
+
+  private globalConfig = {};
+
+  constructor(private http: Http) {
--- End diff --

I think this constructor was carried over from metron-config and the 
property assignments are not needed.  I think it should just be empty.


---

[jira] [Commented] (METRON-1356) Add a mechanism in Java for discovering service host/ports

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474716#comment-16474716
 ] 

ASF GitHub Bot commented on METRON-1356:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/973
  
@ottobackwards just updated the README.  You previously had to lookup the 
docker machine ip address and substitute it in source code to get the tests to 
pass if running locally.  The text I removed in the latest commit described how 
to do that.  So for testing this, you should no longer have to do that and the 
tests should just work.

I haven't heard of that before but I will check it out.


> Add a mechanism in Java for discovering service host/ports
> --
>
> Key: METRON-1356
> URL: https://issues.apache.org/jira/browse/METRON-1356
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Ryan Merriman
>Priority: Major
>
> Integration tests will need to initialize clients with service urls.  These 
> may change depending on where and how the infrastructure is run (Docker 
> engine vs Docker for Mac).  It would be helpful to have a unified way of 
> retrieving these across all integration tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #973: METRON-1356: Add a mechanism in Java for discovering serv...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/973
  
@ottobackwards just updated the README.  You previously had to lookup the 
docker machine ip address and substitute it in source code to get the tests to 
pass if running locally.  The text I removed in the latest commit described how 
to do that.  So for testing this, you should no longer have to do that and the 
tests should just work.

I haven't heard of that before but I will check it out.


---

[jira] [Commented] (METRON-1563) Initial Feature Branch Work

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474687#comment-16474687
 ] 

ASF GitHub Bot commented on METRON-1563:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
I have renamed in preparation for the Feature branch based on the original 
jira ( create a new subtask to land this on feature and change to that ID )


> Initial Feature Branch Work
> ---
>
> Key: METRON-1563
> URL: https://issues.apache.org/jira/browse/METRON-1563
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #687: METRON-1563 [DISCUSS] Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
I have renamed in preparation for the Feature branch based on the original 
jira ( create a new subtask to land this on feature and change to that ID )


---

[jira] [Created] (METRON-1563) Initial Feature Branch Work

[Otto Fowler (JIRA)]

Otto Fowler created METRON-1563:
---

 Summary: Initial Feature Branch Work
 Key: METRON-1563
 URL: https://issues.apache.org/jira/browse/METRON-1563
 Project: Metron
  Issue Type: Sub-task
Reporter: Otto Fowler
Assignee: Otto Fowler






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474648#comment-16474648
 ] 

ASF GitHub Bot commented on METRON-1551:


Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/1012


> Profiler Should Not Use Java Serialization
> --
>
> Key: METRON-1551
> URL: https://issues.apache.org/jira/browse/METRON-1551
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> When running the Profiler in a topology where serialization occurs, the 
> following error happens.  This can occur when the number of workers is 
> greater than 1.
> The topology should not be using Java serialization for serializing tuple 
> values as this will negatively impact performance. 
> {code}
> 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] 
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  ... 6 more
> Caused by: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
> ~[?:1.8.0_162]
>  at 
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
> ~[kryo-3.0.3.jar:?]
>  at 
> 

[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474647#comment-16474647
 ] 

ASF GitHub Bot commented on METRON-1551:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1012
  
Reopen for unrelated failure.  Will open a bug for what seems to be an 
intermittent test failure.
```
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec 
<<< FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest
testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest)  
Time elapsed: 0.031 sec  <<< FAILURE!
java.lang.AssertionError: expected:<6> but was:<5>
```


> Profiler Should Not Use Java Serialization
> --
>
> Key: METRON-1551
> URL: https://issues.apache.org/jira/browse/METRON-1551
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> When running the Profiler in a topology where serialization occurs, the 
> following error happens.  This can occur when the number of workers is 
> greater than 1.
> The topology should not be using Java serialization for serializing tuple 
> values as this will negatively impact performance. 
> {code}
> 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] 
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  ... 6 more
> Caused by: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
> ~[?:1.8.0_162]
>  at 
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]

[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474649#comment-16474649
 ] 

ASF GitHub Bot commented on METRON-1551:


GitHub user nickwallen reopened a pull request:

https://github.com/apache/metron/pull/1012

METRON-1551 Profiler Should Not Use Java Serialization

When running the Profiler in a topology where serialization occurs, the 
following error happens.  This can occur when the number of workers is greater 
than 1.

The topology should not be using Java serialization for serializing tuple 
values as this will negatively impact performance. 

```
2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] 
java.lang.RuntimeException: java.lang.RuntimeException: 
java.io.NotSerializableException: 
org.apache.metron.common.configuration.profiler.ProfileResult
 at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) 
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.RuntimeException: java.io.NotSerializableException: 
org.apache.metron.common.configuration.profiler.ProfileResult
 at 
org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
 ~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
 ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
~[kryo-3.0.3.jar:?]
 at 
org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 ... 6 more
Caused by: java.io.NotSerializableException: 
org.apache.metron.common.configuration.profiler.ProfileResult
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
~[?:1.8.0_162]
 at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
~[?:1.8.0_162]
 at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
~[?:1.8.0_162]
 at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
~[?:1.8.0_162]
 at 
org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
 ~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
 ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
~[kryo-3.0.3.jar:?]
 at 
org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
 

[GitHub] metron pull request #1012: METRON-1551 Profiler Should Not Use Java Serializ...

[nickwallen]

Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/1012


---

[GitHub] metron pull request #1012: METRON-1551 Profiler Should Not Use Java Serializ...

[nickwallen]

GitHub user nickwallen reopened a pull request:

https://github.com/apache/metron/pull/1012

METRON-1551 Profiler Should Not Use Java Serialization

When running the Profiler in a topology where serialization occurs, the 
following error happens.  This can occur when the number of workers is greater 
than 1.

The topology should not be using Java serialization for serializing tuple 
values as this will negatively impact performance. 

```
2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] 
java.lang.RuntimeException: java.lang.RuntimeException: 
java.io.NotSerializableException: 
org.apache.metron.common.configuration.profiler.ProfileResult
 at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) 
~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.RuntimeException: java.io.NotSerializableException: 
org.apache.metron.common.configuration.profiler.ProfileResult
 at 
org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
 ~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
 ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
~[kryo-3.0.3.jar:?]
 at 
org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 ... 6 more
Caused by: java.io.NotSerializableException: 
org.apache.metron.common.configuration.profiler.ProfileResult
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
~[?:1.8.0_162]
 at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
~[?:1.8.0_162]
 at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
~[?:1.8.0_162]
 at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
~[?:1.8.0_162]
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
~[?:1.8.0_162]
 at 
org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
 ~[kryo-3.0.3.jar:?]
 at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
 ~[kryo-3.0.3.jar:?]
 at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
~[kryo-3.0.3.jar:?]
 at 
org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 
org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
 ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
 at 

[GitHub] metron issue #1012: METRON-1551 Profiler Should Not Use Java Serialization

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1012
  
Reopen for unrelated failure.  Will open a bug for what seems to be an 
intermittent test failure.
```
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec 
<<< FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest
testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest)  
Time elapsed: 0.031 sec  <<< FAILURE!
java.lang.AssertionError: expected:<6> but was:<5>
```


---

[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474636#comment-16474636
 ] 

ASF GitHub Bot commented on METRON-1551:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1012
  
Looks good; +1 by inspection pending travis


> Profiler Should Not Use Java Serialization
> --
>
> Key: METRON-1551
> URL: https://issues.apache.org/jira/browse/METRON-1551
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> When running the Profiler in a topology where serialization occurs, the 
> following error happens.  This can occur when the number of workers is 
> greater than 1.
> The topology should not be using Java serialization for serializing tuple 
> values as this will negatively impact performance. 
> {code}
> 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] 
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  ... 6 more
> Caused by: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
> ~[?:1.8.0_162]
>  at 
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at 

[GitHub] metron issue #1012: METRON-1551 Profiler Should Not Use Java Serialization

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1012
  
Looks good; +1 by inspection pending travis


---

[jira] [Commented] (METRON-1554) Pcap Query Panel

[Ryan Merriman (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474591#comment-16474591
 ] 

Ryan Merriman commented on METRON-1554:
---

[~ottobackwards] I believe a lot of the functionality was carried forward and 
is contained in these READMEs:
 * 
[https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend|https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend,]
 * [https://github.com/apache/metron/tree/master/metron-platform/metron-api]

As far as the old UI functionality, the code is spread out in various source 
code files under 
[https://github.com/apache/metron/tree/Metron_0.1BETA/metron-ui.]  I can't find 
any documentation on it though.

> Pcap Query Panel
> 
>
> Key: METRON-1554
> URL: https://issues.apache.org/jira/browse/METRON-1554
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>Priority: Major
>
> Legacy OpenSOC included a panel in Kibana that allowed users to query for 
> pcap data.  We would like to add this feature back into Metron.  There are 2 
> discussions happening on the dev list where we are gathering user 
> requirements:
> [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3CCAEVkqPYxfe3Q65mX7Mkuk_FKUCV420yb6hcLmf+FF=1ozer...@mail.gmail.com%3E]
> and working through the backend architecture:
> [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3ccaevkqpbxzjnu_wgrbfwnz-mvqnkb7mthedveq9plyhwfit7...@mail.gmail.com%3E]
>  Forthcoming sub tasks will be based on the outcome of these discussions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1562) Enable Kerberos in REST for YARN and MR jobs

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1562:
-

 Summary: Enable Kerberos in REST for YARN and MR jobs
 Key: METRON-1562
 URL: https://issues.apache.org/jira/browse/METRON-1562
 Project: Metron
  Issue Type: Sub-task
Reporter: Ryan Merriman


We need to enable Kerberos support in REST for YARN and MR interactions.  This 
will include:
 * Kerberos authentication for YARN REST api
 * Kerberos authentication for submitting MR jobs
 * Kerberos authentication for accessing query results (this is likely already 
done but should be validated)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1561) Add security to Pcap endpoints

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1561:
-

 Summary: Add security to Pcap endpoints
 Key: METRON-1561
 URL: https://issues.apache.org/jira/browse/METRON-1561
 Project: Metron
  Issue Type: Sub-task
Reporter: Ryan Merriman


Pcap endpoints should be secured as follows:
 * users should only be able to view status and access/cleanup results of the 
queries they submit
 * an admin role should be provided that has access to and can perform actions 
on all jobs/queries
 * a map of users and jobs they have submitted should be persisted so that a 
list of a user's jobs can be returned



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1560) Update MPack to support Pcap panel

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1560:
-

 Summary: Update MPack to support Pcap panel
 Key: METRON-1560
 URL: https://issues.apache.org/jira/browse/METRON-1560
 Project: Metron
  Issue Type: Sub-task
Reporter: Ryan Merriman


The MPack will need to be adjusted to support the Pcap panel.  This includes:
 * prompting a user during the MPack installation process to agree to tshark 
being installed
 * installing tshark on the correct node(s)
 * adding pcap related properties to the MPack so they can be managed in Ambari



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1559) Create Pcap Service

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1559:
-

 Summary: Create Pcap Service
 Key: METRON-1559
 URL: https://issues.apache.org/jira/browse/METRON-1559
 Project: Metron
  Issue Type: Sub-task
Reporter: Ryan Merriman


We need a service that exposes the various Pcap endpoints.  These include:
 * GET /api/v1/pcap/metadata?basePath - This endpoint will return metadata of 
pcap data stored in HDFS.  This would include pcap size, date ranges (how far 
back can I go), etc.  It would accept an optional HDFS basePath parameter for 
cases where pcap data is stored in multiple places and/or different from the 
default location.
 * POST /api/v1/pcap/fixed - This endpoint would accept a fixed pcap request, 
submit a pcap job, and return a job id.  The request would be an object 
containing the options documented here for the fixed filter:  
[https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend#query-filter-utility].
  A job will be associated with a user that submits it.  An exception will be 
returned for violating constraints like too many queries submitted, query 
parameters out of limits, etc.  A record of the user and job id will be 
persisted to a data store so a list of a user's jobs can later be retrieved.
 * POST /api/v1/pcap/query - This endpoint would accept a query pcap request, 
submit a pcap job, and return a job id.  The request would be an object 
containing the options documented here for the query filter:  
[https://github.com/apache/metron/tree/master/metron-platform/metron-pcap-backend#query-filter-utility].
  A job will be associated with a user that submits it.  An exception will be 
returned for violating constraints like too many queries submitted, query 
parameters out of limits, etc.  A record of the user and job id will be 
persisted to a data store so a list of a user's jobs can later be retrieved.
 * GET /api/v1/pcap/status/ - This endpoint will return the YARN status 
of a running/completed job.
 * GET /api/v1/pcap/stop/ - This endpoint would kill a running pcap job. 
 If the job has already completed this is a noop.
 * GET /api/v1/pcap/list - This endpoint will list a user's submitted pcap 
queries.  Items in the list would contain job id, status (is it finished?), 
start/end time, and number of pages.
 * GET /api/v1/pcap/pdml// - This endpoint will return pcap 
results for the given page in pdml format ([https://wiki.wireshark.org/PDML]).  
Are there other formats we want to support?
 * GET /api/v1/pcap/raw// - This endpoint will allow a user 
to download raw pcap results for the given page.
 * DELETE /api/v1/pcap/ - This endpoint will delete pcap query results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474572#comment-16474572
 ] 

ASF GitHub Bot commented on METRON-1551:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1012
  
I made two updates. 

* I added the additional test check to `StellarProcessorUtils` to ensure 
the value returned is Java serializable.  There were a few classes that I had 
to update for the tests to continue to pass.  All simple cases where a class 
just needed to implement Serializable.

* Updated all Profiler classes to be Java serializable in case a user 
chooses to use Java serialization in their Storm topology. This is not 
recommended, but there is no reason to block a user from doing so.






> Profiler Should Not Use Java Serialization
> --
>
> Key: METRON-1551
> URL: https://issues.apache.org/jira/browse/METRON-1551
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> When running the Profiler in a topology where serialization occurs, the 
> following error happens.  This can occur when the number of workers is 
> greater than 1.
> The topology should not be using Java serialization for serializing tuple 
> values as this will negatively impact performance. 
> {code}
> 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] 
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  ... 6 more
> Caused by: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 
> ~[?:1.8.0_162]
>  at 
> 

[GitHub] metron issue #1012: METRON-1551 Profiler Should Not Use Java Serialization

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1012
  
I made two updates. 

* I added the additional test check to `StellarProcessorUtils` to ensure 
the value returned is Java serializable.  There were a few classes that I had 
to update for the tests to continue to pass.  All simple cases where a class 
just needed to implement Serializable.

* Updated all Profiler classes to be Java serializable in case a user 
chooses to use Java serialization in their Storm topology. This is not 
recommended, but there is no reason to block a user from doing so.






---

[jira] [Created] (METRON-1558) Enable paging through Pcap result sets

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1558:
-

 Summary: Enable paging through Pcap result sets
 Key: METRON-1558
 URL: https://issues.apache.org/jira/browse/METRON-1558
 Project: Metron
  Issue Type: Sub-task
Reporter: Ryan Merriman


Due to the volume of pcap data and the potential for very large query results, 
the Pcap query process should write results in manageable chunks or files, so 
that a user can page through the results in an efficient manner.

Page size should be configurable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1557) PcapJob should be asynchronous

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1557:
-

 Summary: PcapJob should be asynchronous
 Key: METRON-1557
 URL: https://issues.apache.org/jira/browse/METRON-1557
 Project: Metron
  Issue Type: Sub-task
Reporter: Ryan Merriman


Pcap job submission should be asynchronous.  The PcapJob class should submit 
the job and return the job id as soon as possible rather than blocking and 
waiting on the job to complete.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1556) Create YARN service

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1556:
-

 Summary: Create YARN service
 Key: METRON-1556
 URL: https://issues.apache.org/jira/browse/METRON-1556
 Project: Metron
  Issue Type: Sub-task
Reporter: Ryan Merriman


We need a YARN Service in REST for interacting with the YARN REST api.  The 
service should at least:
 * Return a description of a job including status, start time, end time, and 
any other relevant information given a job id
 * Stop a job given a job id

The 
[StormStatusService|https://github.com/apache/metron/blob/master/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormStatusServiceImpl.java]
 is similar in that it also proxies to a REST API.  

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1555) Update REST to run YARN and MR jobs

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1555:
-

 Summary: Update REST to run YARN and MR jobs
 Key: METRON-1555
 URL: https://issues.apache.org/jira/browse/METRON-1555
 Project: Metron
  Issue Type: Sub-task
Reporter: Ryan Merriman
Assignee: Ryan Merriman


This task involves enabling REST to submit YARN or MR jobs.  We will likely 
need to:
 * update Maven dependencies to include YARN and MR libraries in the classpath 
and resolve any version conflicts
 * update REST start script to include properties required for YARN
 * update the MPack for any additional setup work (create user HDFS directory 
for example) and properties needed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1551) Profiler Should Not Use Java Serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474333#comment-16474333
 ] 

ASF GitHub Bot commented on METRON-1551:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1012
  
Ok, this looks great, one request though.  It appears that, at least for 
the profiler, it is required that objects used in Stellar be Java serializable 
if they aren't listed in that exception list.

We ensure that the objects are kryo serializable 
[here](https://github.com/apache/metron/blob/91a017b6dcefc250bfd67cecf9803cb59015d213/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/StellarProcessorUtils.java#L64)
  Can we also make a blurb that they're java serializable?  I know that 
OnlineStatisticsProvider will not be java serializable due to its dependence on 
t-digest, so maybe make that check optional but on by default?



> Profiler Should Not Use Java Serialization
> --
>
> Key: METRON-1551
> URL: https://issues.apache.org/jira/browse/METRON-1551
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> When running the Profiler in a topology where serialization occurs, the 
> following error happens.  This can occur when the number of workers is 
> greater than 1.
> The topology should not be using Java serialization for serializing tuple 
> values as this will negatively impact performance. 
> {code}
> 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] 
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at 
> org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
>  ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>  ... 6 more
> Caused by: java.io.NotSerializableException: 
> org.apache.metron.common.configuration.profiler.ProfileResult
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) 
> ~[?:1.8.0_162]
>  at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 
> ~[?:1.8.0_162]
>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 
> ~[?:1.8.0_162]
>  at 

[GitHub] metron issue #1012: METRON-1551 Profiler Should Not Use Java Serialization

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1012
  
Ok, this looks great, one request though.  It appears that, at least for 
the profiler, it is required that objects used in Stellar be Java serializable 
if they aren't listed in that exception list.

We ensure that the objects are kryo serializable 
[here](https://github.com/apache/metron/blob/91a017b6dcefc250bfd67cecf9803cb59015d213/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/StellarProcessorUtils.java#L64)
  Can we also make a blurb that they're java serializable?  I know that 
OnlineStatisticsProvider will not be java serializable due to its dependence on 
t-digest, so maybe make that check optional but on by default?



---

[jira] [Commented] (METRON-1489) Retrofit UI tests to run reliably during nightly QE runs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474269#comment-16474269
 ] 

ASF GitHub Bot commented on METRON-1489:


Github user sardell commented on a diff in the pull request:

https://github.com/apache/metron/pull/1004#discussion_r187971664
  
--- Diff: metron-interface/metron-alerts/README.md ---
@@ -126,7 +126,14 @@ The application will be available at http://host:4201 
assuming the port is set t
 
 ## E2E Tests
 
-An expressjs server is available for mocking the elastic search api.
+#Caveats
--- End diff --

Thanks for pointing this out, @nickwallen. Fixed with the latest commit.


> Retrofit UI tests to run reliably during nightly QE runs
> 
>
> Key: METRON-1489
> URL: https://issues.apache.org/jira/browse/METRON-1489
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1004: METRON-1489: Retrofit UI tests to run reliably

[sardell]

Github user sardell commented on a diff in the pull request:

https://github.com/apache/metron/pull/1004#discussion_r187971664
  
--- Diff: metron-interface/metron-alerts/README.md ---
@@ -126,7 +126,14 @@ The application will be available at http://host:4201 
assuming the port is set t
 
 ## E2E Tests
 
-An expressjs server is available for mocking the elastic search api.
+#Caveats
--- End diff --

Thanks for pointing this out, @nickwallen. Fixed with the latest commit.


---

[jira] [Commented] (METRON-1554) Pcap Query Panel

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474247#comment-16474247
 ] 

Otto Fowler commented on METRON-1554:
-

[~rmerriman] are there any materials around the old functionality that we can 
reference?

> Pcap Query Panel
> 
>
> Key: METRON-1554
> URL: https://issues.apache.org/jira/browse/METRON-1554
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>Priority: Major
>
> Legacy OpenSOC included a panel in Kibana that allowed users to query for 
> pcap data.  We would like to add this feature back into Metron.  There are 2 
> discussions happening on the dev list where we are gathering user 
> requirements:
> [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3CCAEVkqPYxfe3Q65mX7Mkuk_FKUCV420yb6hcLmf+FF=1ozer...@mail.gmail.com%3E]
> and working through the backend architecture:
> [http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3ccaevkqpbxzjnu_wgrbfwnz-mvqnkb7mthedveq9plyhwfit7...@mail.gmail.com%3E]
>  Forthcoming sub tasks will be based on the outcome of these discussions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1554) Pcap Query Panel

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1554:
-

 Summary: Pcap Query Panel
 Key: METRON-1554
 URL: https://issues.apache.org/jira/browse/METRON-1554
 Project: Metron
  Issue Type: New Feature
Reporter: Ryan Merriman


Legacy OpenSOC included a panel in Kibana that allowed users to query for pcap 
data.  We would like to add this feature back into Metron.  There are 2 
discussions happening on the dev list where we are gathering user requirements:

[http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3CCAEVkqPYxfe3Q65mX7Mkuk_FKUCV420yb6hcLmf+FF=1ozer...@mail.gmail.com%3E]

and working through the backend architecture:

[http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3ccaevkqpbxzjnu_wgrbfwnz-mvqnkb7mthedveq9plyhwfit7...@mail.gmail.com%3E]

 Forthcoming sub tasks will be based on the outcome of these discussions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1004: METRON-1489: Retrofit UI tests to run reliably

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1004
  
Please fix the formatting issue with the README.  I have mentioned in 
another thread, but you might have missed it.


---

[jira] [Commented] (METRON-1489) Retrofit UI tests to run reliably during nightly QE runs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474187#comment-16474187
 ] 

ASF GitHub Bot commented on METRON-1489:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1004
  
Please fix the formatting issue with the README.  I have mentioned in 
another thread, but you might have missed it.


> Retrofit UI tests to run reliably during nightly QE runs
> 
>
> Key: METRON-1489
> URL: https://issues.apache.org/jira/browse/METRON-1489
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1489) Retrofit UI tests to run reliably during nightly QE runs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474183#comment-16474183
 ] 

ASF GitHub Bot commented on METRON-1489:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1004#discussion_r187948446
  
--- Diff: 
metron-interface/metron-alerts/e2e/alerts-list/meta-alerts/meta-alert.e2e-spec.ts
 ---
@@ -19,139 +19,142 @@
 import { MetronAlertsPage } from '../alerts-list.po';
 import {customMatchers} from '../../matchers/custom-matchers';
 import {LoginPage} from '../../login/login.po';
-import {loadTestData, deleteTestData} from '../../utils/e2e_util';
+import {loadTestData, deleteTestData, createMetaAlertsIndex} from 
'../../utils/e2e_util';
 import {TreeViewPage} from '../tree-view/tree-view.po';
 import {MetronAlertDetailsPage} from 
'../../alert-details/alert-details.po';
 import {MetaAlertPage} from './meta-alert.po';
 import {AlertFacetsPage} from '../alert-filters/alert-filters.po';
 
-describe('meta-alerts workflow', function() {
+describe('Test spec for meta alerts workflow', function() {
   let detailsPage: MetronAlertDetailsPage;
   let tablePage: MetronAlertsPage;
   let metaAlertPage: MetaAlertPage;
   let treePage: TreeViewPage;
   let loginPage: LoginPage;
   let alertFacetsPage: AlertFacetsPage;
 
-  beforeAll(() => {
-loadTestData();
-
+  beforeAll(async function() : Promise {
 loginPage = new LoginPage();
-loginPage.login();
 tablePage = new MetronAlertsPage();
 treePage = new TreeViewPage();
 tablePage = new MetronAlertsPage();
 metaAlertPage = new MetaAlertPage();
 detailsPage = new MetronAlertDetailsPage();
 alertFacetsPage = new AlertFacetsPage();
+
+await createMetaAlertsIndex();
+await loadTestData();
+await loginPage.login();
   });
 
-  afterAll(() => {
-loginPage.logout();
-deleteTestData();
+  afterAll(async function() : Promise {
+await loginPage.logout();
+await deleteTestData();
   });
 
   beforeEach(() => {
 jasmine.addMatchers(customMatchers);
   });
 
-  it('should have all the steps for meta alerts workflow', () => {
+  it('should have all the steps for meta alerts workflow', async 
function() : Promise {
 let comment1 = 'This is a sample comment';
 let userNameAndTimestamp = '- admin - a few seconds ago';
-let confirmText = 'Do you wish to create a meta alert with 113 
selected alerts?';
+let confirmText = 'Do you wish to create a meta alert with 13 selected 
alerts?';
 let dashRowValues = {
-  'firstDashRow': ['0', '192.168.138.158', 'ALERTS', '113'],
-  'secondDashRow': ['0', '192.168.66.1', 'ALERTS', '56']
+  'firstDashRow': ['0', 'runlove.us', 'ALERTS', '13']
 };
 
-tablePage.navigateTo();
+await tablePage.navigateTo();
+expect(await tablePage.getChangesAlertTableTitle('Alerts 
(0)')).toEqual('Alerts (169)');
 
 /* Create Meta Alert */
-treePage.selectGroup('ip_src_addr');
-
expect(treePage.getDashGroupValues('192.168.138.158')).toEqualBcoz(dashRowValues.firstDashRow,
 'First Dashrow to be present');
-
expect(treePage.getDashGroupValues('192.168.66.1')).toEqualBcoz(dashRowValues.secondDashRow,
 'Second Dashrow to be present');
+await treePage.selectGroup('host');
+expect(await 
treePage.getDashGroupValues('runlove.us')).toEqualBcoz(dashRowValues.firstDashRow,
 'First Dashrow to be present');
 
-treePage.clickOnMergeAlerts('192.168.138.158');
-expect(treePage.getConfirmationText()).toEqualBcoz(confirmText, 
'confirmation text to be present');
-treePage.clickNoForConfirmation();
+await treePage.clickOnMergeAlerts('runlove.us');
+expect(await treePage.getConfirmationText()).toEqualBcoz(confirmText, 
'confirmation text to be present');
+await treePage.clickNoForConfirmation();
 
-treePage.clickOnMergeAlerts('192.168.138.158');
-treePage.clickYesForConfirmation();
+await treePage.clickOnMergeAlerts('runlove.us');
+await treePage.clickYesForConfirmation();
 
-treePage.waitForElementToDisappear('192.168.138.158');
+await treePage.waitForElementToDisappear('runlove.us');
 
-treePage.unGroup();
+await treePage.unGroup();
 
 /* Table should have all alerts */
-tablePage.waitForMetaAlert();
-expect(tablePage.getPaginationText()).toEqualBcoz('1 - 25 of 57', 
'pagination text to be present');
-expect(tablePage.getCellValue(0, 2, '(114)')).toContain('(113)', 
'number of alerts in a meta alert should be correct');
-

[GitHub] metron pull request #1004: METRON-1489: Retrofit UI tests to run reliably

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1004#discussion_r187948446
  
--- Diff: 
metron-interface/metron-alerts/e2e/alerts-list/meta-alerts/meta-alert.e2e-spec.ts
 ---
@@ -19,139 +19,142 @@
 import { MetronAlertsPage } from '../alerts-list.po';
 import {customMatchers} from '../../matchers/custom-matchers';
 import {LoginPage} from '../../login/login.po';
-import {loadTestData, deleteTestData} from '../../utils/e2e_util';
+import {loadTestData, deleteTestData, createMetaAlertsIndex} from 
'../../utils/e2e_util';
 import {TreeViewPage} from '../tree-view/tree-view.po';
 import {MetronAlertDetailsPage} from 
'../../alert-details/alert-details.po';
 import {MetaAlertPage} from './meta-alert.po';
 import {AlertFacetsPage} from '../alert-filters/alert-filters.po';
 
-describe('meta-alerts workflow', function() {
+describe('Test spec for meta alerts workflow', function() {
   let detailsPage: MetronAlertDetailsPage;
   let tablePage: MetronAlertsPage;
   let metaAlertPage: MetaAlertPage;
   let treePage: TreeViewPage;
   let loginPage: LoginPage;
   let alertFacetsPage: AlertFacetsPage;
 
-  beforeAll(() => {
-loadTestData();
-
+  beforeAll(async function() : Promise {
 loginPage = new LoginPage();
-loginPage.login();
 tablePage = new MetronAlertsPage();
 treePage = new TreeViewPage();
 tablePage = new MetronAlertsPage();
 metaAlertPage = new MetaAlertPage();
 detailsPage = new MetronAlertDetailsPage();
 alertFacetsPage = new AlertFacetsPage();
+
+await createMetaAlertsIndex();
+await loadTestData();
+await loginPage.login();
   });
 
-  afterAll(() => {
-loginPage.logout();
-deleteTestData();
+  afterAll(async function() : Promise {
+await loginPage.logout();
+await deleteTestData();
   });
 
   beforeEach(() => {
 jasmine.addMatchers(customMatchers);
   });
 
-  it('should have all the steps for meta alerts workflow', () => {
+  it('should have all the steps for meta alerts workflow', async 
function() : Promise {
 let comment1 = 'This is a sample comment';
 let userNameAndTimestamp = '- admin - a few seconds ago';
-let confirmText = 'Do you wish to create a meta alert with 113 
selected alerts?';
+let confirmText = 'Do you wish to create a meta alert with 13 selected 
alerts?';
 let dashRowValues = {
-  'firstDashRow': ['0', '192.168.138.158', 'ALERTS', '113'],
-  'secondDashRow': ['0', '192.168.66.1', 'ALERTS', '56']
+  'firstDashRow': ['0', 'runlove.us', 'ALERTS', '13']
 };
 
-tablePage.navigateTo();
+await tablePage.navigateTo();
+expect(await tablePage.getChangesAlertTableTitle('Alerts 
(0)')).toEqual('Alerts (169)');
 
 /* Create Meta Alert */
-treePage.selectGroup('ip_src_addr');
-
expect(treePage.getDashGroupValues('192.168.138.158')).toEqualBcoz(dashRowValues.firstDashRow,
 'First Dashrow to be present');
-
expect(treePage.getDashGroupValues('192.168.66.1')).toEqualBcoz(dashRowValues.secondDashRow,
 'Second Dashrow to be present');
+await treePage.selectGroup('host');
+expect(await 
treePage.getDashGroupValues('runlove.us')).toEqualBcoz(dashRowValues.firstDashRow,
 'First Dashrow to be present');
 
-treePage.clickOnMergeAlerts('192.168.138.158');
-expect(treePage.getConfirmationText()).toEqualBcoz(confirmText, 
'confirmation text to be present');
-treePage.clickNoForConfirmation();
+await treePage.clickOnMergeAlerts('runlove.us');
+expect(await treePage.getConfirmationText()).toEqualBcoz(confirmText, 
'confirmation text to be present');
+await treePage.clickNoForConfirmation();
 
-treePage.clickOnMergeAlerts('192.168.138.158');
-treePage.clickYesForConfirmation();
+await treePage.clickOnMergeAlerts('runlove.us');
+await treePage.clickYesForConfirmation();
 
-treePage.waitForElementToDisappear('192.168.138.158');
+await treePage.waitForElementToDisappear('runlove.us');
 
-treePage.unGroup();
+await treePage.unGroup();
 
 /* Table should have all alerts */
-tablePage.waitForMetaAlert();
-expect(tablePage.getPaginationText()).toEqualBcoz('1 - 25 of 57', 
'pagination text to be present');
-expect(tablePage.getCellValue(0, 2, '(114)')).toContain('(113)', 
'number of alerts in a meta alert should be correct');
-expect(tablePage.getNonHiddenRowCount()).toEqualBcoz(25, '25 rows to 
be visible');
-expect(tablePage.getAllRowsCount()).toEqualBcoz(138, '138 rows to be 
available');
-expect(tablePage.getHiddenRowCount()).toEqualBcoz(113, 

[jira] [Commented] (METRON-1065) Grok pattern for Cisco ASA Parser expects syslog_pri

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474170#comment-16474170
 ] 

ASF GitHub Bot commented on METRON-1065:


Github user Humbedooh closed the pull request at:

https://github.com/apache/metron/pull/670


> Grok pattern for Cisco ASA Parser expects syslog_pri
> 
>
> Key: METRON-1065
> URL: https://issues.apache.org/jira/browse/METRON-1065
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.1
>Reporter: Bas van de Lustgraaf
>Priority: Minor
>
> The current grok pattern `CISCO_TAGGED_SYSLOG` expects to have a syslog 
> priority present at the start of each message. Unfortunately, this is not 
> always the case.
> *Currently supported:*
> {noformat}
> <162>Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from 
> 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside
> {noformat}
> *Not supported by the current Grok pattern:*
> {noformat}
> Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from 
> 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside
> {noformat}
> My suggestion would be to edit the `CISCO_TAGGED_SYSLOG` pattern to make the 
> following part optional: 
> {noformat}
> <%{POSINT:syslog_pri}>
> {noformat}
> And grep the severity from the `%ASA-4-106023` part. The part between the 
> hyphens, is the severity (source 
> http://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs-sev-level.html).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1065) Grok pattern for Cisco ASA Parser expects syslog_pri

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474169#comment-16474169
 ] 

ASF GitHub Bot commented on METRON-1065:


GitHub user basvdl reopened a pull request:

https://github.com/apache/metron/pull/670

METRON-1065 grok pattern for cisco asa parser expects syslog pri

## Contributor Comments
Cisco ASA logs does not always contain a priority code at the beginnen of 
the event. Therefore I've changed the grok pattern to make this optional.

`<164>Aug 05 2016 01:01:34: %ASA-4-106023: Deny tcp src 
Inside:10.30.9.121/54580 dst Outside:192.168.135.51/42028 by access-group 
"Inside_access_in" [0x962df600, 0x0]`

<164> should be optional.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated **unit tests** and or integration tests 
to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/basvdl/metron 
METRON-1065_Grok_pattern_for_Cisco_ASA_Parser_expects_syslog_pri

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/670.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #670


commit c691e3374ea9e9bdb4c291071adae045516e744b
Author: Bas van de Lustgraaf 
Date:   2017-07-26T14:04:05Z

Priority for Cisco ASA logging is optional

commit a731004e53b948979bdbb6d01bdd274702eef0c7
Author: Bas van de Lustgraaf 
Date:   2017-07-26T14:10:14Z

Fixed formatting

commit 69584cdee5098bcc6449c72f5a69129e20c23cd8
Author: Bas van de Lustgraaf 
Date:   2017-07-26T14:18:04Z

Fix the license header after using the intelliJ auto format




> Grok pattern for Cisco ASA Parser expects syslog_pri
> 
>
> Key: METRON-1065
> URL: https://issues.apache.org/jira/browse/METRON-1065
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.1
>Reporter: Bas van de Lustgraaf
>Priority: Minor
>
> The current grok pattern `CISCO_TAGGED_SYSLOG` 

[jira] [Commented] (METRON-295) Script parsing bolt

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474168#comment-16474168
 ] 

ASF GitHub Bot commented on METRON-295:
---

Github user Humbedooh closed the pull request at:

https://github.com/apache/metron/pull/338


> Script parsing bolt 
> 
>
> Key: METRON-295
> URL: https://issues.apache.org/jira/browse/METRON-295
> Project: Metron
>  Issue Type: New Feature
>Affects Versions: 0.2.2BETA
>Reporter: James Sirota
>Assignee: Karthik Narayanan
>Priority: Minor
>  Labels: newbie, platform
>
> In addition to having a Grok parsing bolt we need a bolt that can execute a 
> script in order to parse a telemetry.  This way you can still script the 
> parsing for telemetries for which Grok expressions are too complex, but still 
> don't have to define a java parser 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #338: METRON-295: Script parsing bolt

[Humbedooh]

Github user Humbedooh closed the pull request at:

https://github.com/apache/metron/pull/338


---

[GitHub] metron pull request #670: METRON-1065 grok pattern for cisco asa parser expe...

[Humbedooh]

Github user Humbedooh closed the pull request at:

https://github.com/apache/metron/pull/670


---

[GitHub] metron pull request #670: METRON-1065 grok pattern for cisco asa parser expe...

[basvdl]

GitHub user basvdl reopened a pull request:

https://github.com/apache/metron/pull/670

METRON-1065 grok pattern for cisco asa parser expects syslog pri

## Contributor Comments
Cisco ASA logs does not always contain a priority code at the beginnen of 
the event. Therefore I've changed the grok pattern to make this optional.

`<164>Aug 05 2016 01:01:34: %ASA-4-106023: Deny tcp src 
Inside:10.30.9.121/54580 dst Outside:192.168.135.51/42028 by access-group 
"Inside_access_in" [0x962df600, 0x0]`

<164> should be optional.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated **unit tests** and or integration tests 
to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/basvdl/metron 
METRON-1065_Grok_pattern_for_Cisco_ASA_Parser_expects_syslog_pri

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/670.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #670


commit c691e3374ea9e9bdb4c291071adae045516e744b
Author: Bas van de Lustgraaf 
Date:   2017-07-26T14:04:05Z

Priority for Cisco ASA logging is optional

commit a731004e53b948979bdbb6d01bdd274702eef0c7
Author: Bas van de Lustgraaf 
Date:   2017-07-26T14:10:14Z

Fixed formatting

commit 69584cdee5098bcc6449c72f5a69129e20c23cd8
Author: Bas van de Lustgraaf 
Date:   2017-07-26T14:18:04Z

Fix the license header after using the intelliJ auto format




---

[jira] [Commented] (METRON-1065) Grok pattern for Cisco ASA Parser expects syslog_pri

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474151#comment-16474151
 ] 

ASF GitHub Bot commented on METRON-1065:


Github user basvdl closed the pull request at:

https://github.com/apache/metron/pull/670


> Grok pattern for Cisco ASA Parser expects syslog_pri
> 
>
> Key: METRON-1065
> URL: https://issues.apache.org/jira/browse/METRON-1065
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.1
>Reporter: Bas van de Lustgraaf
>Priority: Minor
>
> The current grok pattern `CISCO_TAGGED_SYSLOG` expects to have a syslog 
> priority present at the start of each message. Unfortunately, this is not 
> always the case.
> *Currently supported:*
> {noformat}
> <162>Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from 
> 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside
> {noformat}
> *Not supported by the current Grok pattern:*
> {noformat}
> Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from 
> 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside
> {noformat}
> My suggestion would be to edit the `CISCO_TAGGED_SYSLOG` pattern to make the 
> following part optional: 
> {noformat}
> <%{POSINT:syslog_pri}>
> {noformat}
> And grep the severity from the `%ASA-4-106023` part. The part between the 
> hyphens, is the severity (source 
> http://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs-sev-level.html).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #670: METRON-1065 grok pattern for cisco asa parser expe...

[basvdl]

Github user basvdl closed the pull request at:

https://github.com/apache/metron/pull/670


---

[jira] [Commented] (METRON-1065) Grok pattern for Cisco ASA Parser expects syslog_pri

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474148#comment-16474148
 ] 

ASF GitHub Bot commented on METRON-1065:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/670
  
-1 

There has been no response from the contributor.  This PR will be closed 
per the [Metron Development 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
 on Inactive PRs



> Grok pattern for Cisco ASA Parser expects syslog_pri
> 
>
> Key: METRON-1065
> URL: https://issues.apache.org/jira/browse/METRON-1065
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.1
>Reporter: Bas van de Lustgraaf
>Priority: Minor
>
> The current grok pattern `CISCO_TAGGED_SYSLOG` expects to have a syslog 
> priority present at the start of each message. Unfortunately, this is not 
> always the case.
> *Currently supported:*
> {noformat}
> <162>Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from 
> 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside
> {noformat}
> *Not supported by the current Grok pattern:*
> {noformat}
> Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from 
> 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside
> {noformat}
> My suggestion would be to edit the `CISCO_TAGGED_SYSLOG` pattern to make the 
> following part optional: 
> {noformat}
> <%{POSINT:syslog_pri}>
> {noformat}
> And grep the severity from the `%ASA-4-106023` part. The part between the 
> hyphens, is the severity (source 
> http://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs-sev-level.html).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #670: METRON-1065 grok pattern for cisco asa parser expects sys...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/670
  
-1 

There has been no response from the contributor.  This PR will be closed 
per the [Metron Development 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
 on Inactive PRs



---

[jira] [Commented] (METRON-295) Script parsing bolt

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474146#comment-16474146
 ] 

ASF GitHub Bot commented on METRON-295:
---

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/338
  
-1

There has been no response from contributor.  This PR will be closed 
according to the [Metron Development 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
 on Inactive PRs.


> Script parsing bolt 
> 
>
> Key: METRON-295
> URL: https://issues.apache.org/jira/browse/METRON-295
> Project: Metron
>  Issue Type: New Feature
>Affects Versions: 0.2.2BETA
>Reporter: James Sirota
>Assignee: Karthik Narayanan
>Priority: Minor
>  Labels: newbie, platform
>
> In addition to having a Grok parsing bolt we need a bolt that can execute a 
> script in order to parse a telemetry.  This way you can still script the 
> parsing for telemetries for which Grok expressions are too complex, but still 
> don't have to define a java parser 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #338: METRON-295: Script parsing bolt

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/338
  
-1

There has been no response from contributor.  This PR will be closed 
according to the [Metron Development 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
 on Inactive PRs.


---

[jira] [Commented] (METRON-451) SerDeUtils - java.lang.ClassNotFoundException: org.apache.storm.shade.org.joda.time.DateTime

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474143#comment-16474143
 ] 

ASF GitHub Bot commented on METRON-451:
---

Github user Humbedooh closed the pull request at:

https://github.com/apache/metron/pull/278


> SerDeUtils - java.lang.ClassNotFoundException: 
> org.apache.storm.shade.org.joda.time.DateTime
> 
>
> Key: METRON-451
> URL: https://issues.apache.org/jira/browse/METRON-451
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> Ran into this issue attempting to write profile data to HBase.
> {code}
> Exception in thread "main" java.lang.IllegalStateException: Unable to 
> serialize 4790 because org/apache/storm/shade/org/joda/time/DateTime
>   at org.apache.metron.common.utils.SerDeUtils.toBytes(SerDeUtils.java:233)
>   at 
> org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder.columns(ValueOnlyColumnBuilder.java:52)
>   at 
> org.apache.metron.profiler.client.hbase.HBaseProfileWriter.write(HBaseProfileWriter.java:92)
>   at 
> org.apache.metron.profiler.client.hbase.HBaseProfileWriter.write(HBaseProfileWriter.java:80)
>   at 
> org.apache.metron.profiler.client.hbase.ProfilerWriterCLI.main(ProfilerWriterCLI.java:149)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> Caused by: java.lang.NoClassDefFoundError: 
> org/apache/storm/shade/org/joda/time/DateTime
>   at 
> org.apache.metron.common.utils.SerDeUtils$1.initialValue(SerDeUtils.java:87)
>   at 
> org.apache.metron.common.utils.SerDeUtils$1.initialValue(SerDeUtils.java:63)
>   at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
>   at java.lang.ThreadLocal.get(ThreadLocal.java:170)
>   at org.apache.metron.common.utils.SerDeUtils.toBytes(SerDeUtils.java:226)
>   ... 9 more
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.storm.shade.org.joda.time.DateTime
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 14 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #278: Metron 451 SerDeUtils - java.lang.ClassNotFoundExc...

[Humbedooh]

Github user Humbedooh closed the pull request at:

https://github.com/apache/metron/pull/278


---

[GitHub] metron issue #278: Metron 451 SerDeUtils - java.lang.ClassNotFoundException:...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/278
  
-1 Based on the [Metron Development 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
 on Inactive PR.  Missing contributor feedback.  I will open an Apache Infra 
request to close this PR.


---

[jira] [Commented] (METRON-1548) Alerts UI: Remove hardcoded source:type and other fields that may vary

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474122#comment-16474122
 ] 

ASF GitHub Bot commented on METRON-1548:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1010
  
Thanks for the contribution, getting this cleaned up is really helpful!

Could you update 
https://github.com/apache/metron/tree/master/metron-platform/metron-common#global-configuration
 with the new config property?

I'd also like to see the property be something like `source.type.field` to 
be a bit more explanatory and consistent with the other non-Object type fields.


> Alerts UI: Remove hardcoded source:type and other fields that may vary
> --
>
> Key: METRON-1548
> URL: https://issues.apache.org/jira/browse/METRON-1548
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Priority: Major
>
> In Solr, we use source.type instead of source:type (which was originally due 
> to ES limitations, we'd prefer it to be common across both). However, it's 
> hardcoded in the alerts UI to be source:type, so nothing in the UI has a 
> source type and it breaks other things like metaalerts (which rely on source 
> type for being able to pull the data together).
> Any other configs / queries that would cause similar problems should be 
> updated appropriately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1010: METRON-1548: Remove hardcoded source:type from Alerts UI

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1010
  
Thanks for the contribution, getting this cleaned up is really helpful!

Could you update 
https://github.com/apache/metron/tree/master/metron-platform/metron-common#global-configuration
 with the new config property?

I'd also like to see the property be something like `source.type.field` to 
be a bit more explanatory and consistent with the other non-Object type fields.


---