[GitHub] metron pull request #528: METRON-838 Incorrect set of ts in FireEye parser

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/528#discussion_r131635959
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/fireeye/BasicFireEyeParserTest.java
 ---
@@ -57,4 +62,16 @@ public void testParse() throws ParseException {
   }
 }
   }
+
+  private final static String fireeyeMessage = "<164>Mar 19 05:24:39 
10.220.15.15 fenotify-851983.alert: 
CEF:0|FireEye|CMS|7.2.1.244420|DM|domain-match|1|rt=Feb 09 2015 12:28:26 UTC 
dvc=10.201.78.57 cn3Label=cncPort cn3=53 cn2Label=sid cn2=80494706 
shost=dev001srv02.example.com proto=udp cs5Label=cncHost cs5=mfdclk001.org 
dvchost=DEVFEYE1 spt=54527 dvc=10.100.25.16 smac=00:00:0c:07:ac:00 
cn1Label=vlan cn1=0 externalId=851983 cs4Label=link 
cs4=https://DEVCMS01.example.com/event_stream/events_for_bot?ev_id\\=851983 
dmac=00:1d:a2:af:32:a1 cs1Label=sname cs1=Trojan.Generic.DNS";
+
+  @SuppressWarnings("rawtypes")
+  @Test
+  public void testTimestampParsing() throws ParseException {
+JSONObject parsed = parser.parse(fireeyeMessage.getBytes()).get(0);
+JSONParser parser = new JSONParser();
+Map json = (Map) parser.parse(parsed.toJSONString());
+long expectedTimestamp = ZonedDateTime.of(Year.now(UTC).getValue(), 3, 
19, 5, 24, 39, 0, UTC).toInstant().toEpochMilli();
--- End diff --

It's incredibly minor (and optional), but we could just swap out the 
`ZoneId.of("UTC")` for `ZoneOffset.UTC`

At that point, this changes slightly, but still seems reasonable
```
long expectedTimestamp = ZonedDateTime.of(
Year.now(ZoneOffset.UTC).getValue(),
3,
19,
5,
24,
39,
0,
ZoneOffset.UTC
).toInstant().toEpochMilli();
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #528: METRON-838 Incorrect set of ts in FireEye parser

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/528
  
+1 to this.  There's a review comment, but quite frankly I consider it 
optional.  If @bjigmp doesn't want to make that change, I'm still good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Upgrade vagrant base to centos 7

[David Lyle]

Hi Jon,

I'm not sure how to answer your question, but the base image (used for
full-dev) we use is defined here:
https://github.com/apache/metron/blob/master/metron-deployment/packaging/packer-build/base-centos-6.7.json.
It's forked from  bento/centos6.7, but not derived from it. Same deal with
quick-dev.

Readme for building the boxes is here:
https://github.com/apache/metron/blob/master/metron-deployment/packaging/packer-build/README.md
.

-D...


On Sun, Aug 6, 2017 at 10:34 AM, Otto Fowler 
wrote:

> https://issues.apache.org/jira/browse/METRON-667
>
>
>
> On August 6, 2017 at 08:54:50, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> I'm working on a few updates/changes to the bro portions of Metron, in
> preparation for BroCon  in
> September. I'm running into a couple of dependency issues which would be
> most cleanly resolved by a move to a centos 7 base, so I was going to take
> on METRON-559  first,
> before moving on METRON-1088
>  and METRON-813
> .
>
> It looks like as of 3/10
> <
> https://github.com/apache/metron/commit/68a334a8717d2a6b82f7f9651c57bc
> 75e256ba97>
>
> we cut over from using bento on full-dev to using our own base vagrant
> box. Do we have any information regarding what was run on that image, so I
> could attempt to migrate those changes to a new base image? I saw in the
> commentary of the PR that maybe this is just a version pinned bento
> centos6.7 box with some minor swap space changes?
>
> Thanks,
>
> Jon
> --
>
> Jon
>

[GitHub] metron issue #680: METRON-1078: Metron Indexing fails to stop during "Stop A...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/metron/pull/680
  
@justinleet - this seems better. The change is to not error when the rest 
service is unavailable but use that as an indication that the topologies are 
stopped. The rest is code formatting because I've loaded the checkstyle 
template. :)

I'm not 100% happy with how this works, but wanted to get a quick-ish fix 
out while I got 100% happy. That work is described here: 
[METRON-1093](https://issues.apache.org/jira/browse/METRON-1093)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Upgrade vagrant base to centos 7

[zeo...@gmail.com]

Ahh, perfect, I didn't know we stored that in the repo too.  I will take a
look thanks.

Jon

On Mon, Aug 7, 2017 at 8:18 AM David Lyle  wrote:

> Hi Jon,
>
> I'm not sure how to answer your question, but the base image (used for
> full-dev) we use is defined here:
>
> https://github.com/apache/metron/blob/master/metron-deployment/packaging/packer-build/base-centos-6.7.json
> .
> It's forked from  bento/centos6.7, but not derived from it. Same deal with
> quick-dev.
>
> Readme for building the boxes is here:
>
> https://github.com/apache/metron/blob/master/metron-deployment/packaging/packer-build/README.md
> .
>
> -D...
>
>
> On Sun, Aug 6, 2017 at 10:34 AM, Otto Fowler 
> wrote:
>
> > https://issues.apache.org/jira/browse/METRON-667
> >
> >
> >
> > On August 6, 2017 at 08:54:50, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
> >
> > I'm working on a few updates/changes to the bro portions of Metron, in
> > preparation for BroCon 
> in
> > September. I'm running into a couple of dependency issues which would be
> > most cleanly resolved by a move to a centos 7 base, so I was going to
> take
> > on METRON-559  first,
> > before moving on METRON-1088
> >  and METRON-813
> > .
> >
> > It looks like as of 3/10
> > <
> > https://github.com/apache/metron/commit/68a334a8717d2a6b82f7f9651c57bc
> > 75e256ba97>
> >
> > we cut over from using bento on full-dev to using our own base vagrant
> > box. Do we have any information regarding what was run on that image, so
> I
> > could attempt to migrate those changes to a new base image? I saw in the
> > commentary of the PR that maybe this is just a version pinned bento
> > centos6.7 box with some minor swap space changes?
> >
> > Thanks,
> >
> > Jon
> > --
> >
> > Jon
> >
>
-- 

Jon

[GitHub] metron pull request #683: METRON-1084: Management UI web server license shou...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/683#discussion_r131651567
  
--- Diff: metron-interface/metron-config/scripts/package.json ---
@@ -18,5 +18,5 @@
   },
   "private": true,
   "author": "",
-  "license": "MIT"
+  "license": "Apache 2.0"
--- End diff --

Looking at https://docs.npmjs.com/files/package.json and 
https://spdx.org/licenses/, it looks like this should probably be `Apache-2.0` 
(with the dash).  Is my understanding of this correct?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #683: METRON-1084: Management UI web server license shou...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/683#discussion_r131652946
  
--- Diff: metron-interface/metron-config/scripts/package.json ---
@@ -18,5 +18,5 @@
   },
   "private": true,
   "author": "",
-  "license": "MIT"
+  "license": "Apache 2.0"
--- End diff --

Done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #683: METRON-1084: Management UI web server license should be A...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/683
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #528: METRON-838 Incorrect set of ts in FireEye parser

[bjigmp]

Github user bjigmp commented on the issue:

https://github.com/apache/metron/pull/528
  
Thanks @justinleet. Changed to ZoneOffset.UTC


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #673: METRON-1069: Ambari MPack documentation around developmen...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/673
  
@cestella Is there anything else you'd like to see here now that a couple 
more commits were added?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra The end-to-end tests work for me with the latest commits.  Not 
sure if anything changed since I last ran them.  But yay!

Have you made any progress on licenses?  There are a few libraries that we 
are pulling in that are specifically disallowed by Apache.  I outlined these in 
a previous comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #673: METRON-1069: Ambari MPack documentation around developmen...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/673
  
Nope, +1 by inspection.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #683: METRON-1084: Management UI web server license should be A...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/683
  
@justinleet there are actually 2 package.json files in the management UI 
module.  I just fixed the other one too.  Can you check it out again?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #683: METRON-1084: Management UI web server license should be A...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/683
  
@merrimanr Still +1, thanks for taking care of it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #673: METRON-1069: Ambari MPack documentation around dev...

[justinleet]

Github user justinleet closed the pull request at:

https://github.com/apache/metron/pull/673


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #673: METRON-1069: Ambari MPack documentation around developmen...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/673
  
Closed via manually rebased commits to maintain attribution.

Commits are:
3d4751
c7d8b7
ed689c


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #687: METRON-1090 [NO MERGE UNTIL METRON-711] Add Assignment to...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
Added pre and post increment and decrement


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #666: METRON-1051: Enable the ability to update indexed ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/666


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
I just noticed that you had addressed my findings around the plugin version 
as well. So I think going back to 3.0.1 is ok for now. We do need to figure out 
what is going on with Mac OS before getting this through, as this is a major 
platform for developers (myself included) to work from.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #626: METRON-947 [DO NOT MERGE UNTIL METRON-942] Stellar Manage...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/626
  
I am closing this PR, I'll re-submit when METRON-942 is landed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #626: METRON-947 [DO NOT MERGE UNTIL METRON-942] Stellar...

[ottobackwards]

Github user ottobackwards closed the pull request at:

https://github.com/apache/metron/pull/626


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/620
  
If we just focus on the production dependencies, which AFAIK is all we need 
to worry about, then I think the licenses are not a problem.

```
$ license-checker --production | grep licenses | sort | uniq -c | sort -n
   1├─ licenses: MIT
   1 │  ├─ licenses: (OFL-1.1 AND MIT)
   1 │  ├─ licenses: BSD*
   1 │  ├─ licenses: UNLICENSED
   3 │  ├─ licenses: Apache-2.0
  13 │  ├─ licenses: MIT
```

The single `UNLICENSED` dependency is actually `metron-alerts` itself.  Can 
you correct that?  

```
├─ metron-alerts@0.4.0
│  ├─ licenses: UNLICENSED
│  ├─ private: true
│  ├─ licenseFile: 
/Users/nallen/tmp/metron-pr620/metron-interface/metron-alerts/README.md
│  └─ noticeFile: 
/Users/nallen/tmp/metron-pr620/metron-interface/metron-alerts/NOTICE
```




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/620
  
These lic. check commands needs to make their way into the build, like we 
do with the java stuff. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #688: METRON-1094: MaaS will not start due to classpath ...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/688

METRON-1094: MaaS will not start due to classpath error relating to Yarn

## Contributor Comments
The MaaS service won't start with an stack trace:
```
java.lang.NoSuchMethodError: 
org.apache.hadoop.io.retry.RetryPolicies.retryOtherThanRemoteException(Lorg/apache/hadoop/io/retry/RetryPolicy;Ljava/util/Map;)Lorg/apache/hadoop/io/retry/RetryPolicy;
at 
org.apache.hadoop.yarn.client.RMProxy.createRetryPolicy(RMProxy.java:304)
at 
org.apache.hadoop.yarn.client.RMProxy.createRetryPolicy(RMProxy.java:211)
at 
org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:93)
at 
org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)
at 
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.serviceStart(AMRMClientImpl.java:186)
at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.serviceStart(AMRMClientAsyncImpl.java:96)
at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at 
org.apache.metron.maas.service.ApplicationMaster.run(ApplicationMaster.java:457)
at 
org.apache.metron.maas.service.ApplicationMaster.main(ApplicationMaster.java:312)
```

This is due to us packaging up yarn libraries with the maas service app 
rather than picking them up from Yarn.

You can test this by executing the example detailed 
[here](https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service#example).

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-1094

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/688.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #688


commit 45b07e68988fcca13ea7a54ac82415b78fa987ba
Author: cstella 
Date:   2017-08-07T21:39:31Z

METRON-1

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
Running the generate command in CentOS results in defaulting to plugin 
version 3.0.1 when run from a directory without a pom. In Mac, I'm getting 2.4 
for some reason. I haven't found any settings yet that would change this. I'm 
still investigating. If anyone has any ideas, I'd love to hear them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---