date:20171208

Re: [DISCUSS] Upcoming Release

2017-12-08 Thread Matt Foley

Hah, here it is: https://github.com/apache/metron/pull/743
“This problem seems to only reproduce when one unrolls a tarball rather than 
cloning from github.”

Heh, the exclusion at https://github.com/apache/metron/blob/master/pom.xml#L351 
is still there, but the hashcode in the bundle.css file name has changed from 
a0b6b99c10d9a13dc67e to f56deed131e58bd7ee04.  Sigh.  Did the version of Font 
Awesome fonts change?

On 12/8/17, 5:26 PM, "Matt Foley"  wrote:

I remember having trouble with this bundle.css file on the last release, 
but I can’t remember what we did about it.  Anybody?

On 12/8/17, 1:41 PM, "Otto Fowler"  wrote:

Steps

   - Downloaded tar.gz’s, asc files and KEYS
   - Verified signing of both tar.gz’s
   - searched for rouge 0.4.1 entries
   - verified the main pom.xml
   - built :

mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
2C surefire:test@unit-tests && time mvn -q
surefire:test@integration-tests  && time mvn -q test --projects
metron-interface/metron-config && time build_utils/verify_licenses.sh

Found rat error:

*
Summary
---
Generated at: 2017-12-08T16:33:27-05:00

Notes: 3
Binaries: 193
Archives: 0
Standards: 75

Apache Licensed: 74
Generated Documents: 0

JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.

1 Unknown Licenses

*

Files with unapproved licenses:

/Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css

*

*
Summary
---
Generated at: 2017-12-08T16:33:27-05:00

Notes: 3
Binaries: 193
Archives: 0
Standards: 75

Apache Licensed: 74
Generated Documents: 0

JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.

1 Unknown Licenses

*

Files with unapproved licenses:

/Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css

*

On December 8, 2017 at 04:34:24, Matt Foley (ma...@apache.org) wrote:

Colleagues,
I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/

Given the complexity of this RC, I’d appreciate if a couple people 
would be
willing to kick the tires before we put it up for a vote.

I will myself be going thru the Verify Build process this weekend, as I
won’t be able to do it Friday.

Thanks,
--Matt

On 12/4/17, 2:05 PM, "zeo...@gmail.com"  wrote:

Can we resolve the conversation regarding the second repo? I was waiting
to get more input/preferences from people There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I 
just
need to merge it.

Jon

On Mon, Dec 4, 2017, 17:01 Casey Stella  wrote:

> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley  wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend. Shall I go ahead 
and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before
0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a 
release
> of
> > the current state of the metron-bro-plugin-kafka. I will continue 
the
> > discussion in that thread as to the process for accomplishing that, 
but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley"  wrote:
> >
> > Hope everyone (at least in the U.S.) had a

Re: [DISCUSS] Upcoming Release

2017-12-08 Thread Matt Foley

I remember having trouble with this bundle.css file on the last release, but I 
can’t remember what we did about it.  Anybody?

On 12/8/17, 1:41 PM, "Otto Fowler"  wrote:

Steps

   - Downloaded tar.gz’s, asc files and KEYS
   - Verified signing of both tar.gz’s
   - searched for rouge 0.4.1 entries
   - verified the main pom.xml
   - built :

mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
2C surefire:test@unit-tests && time mvn -q
surefire:test@integration-tests  && time mvn -q test --projects
metron-interface/metron-config && time build_utils/verify_licenses.sh

Found rat error:


*
Summary
---
Generated at: 2017-12-08T16:33:27-05:00

Notes: 3
Binaries: 193
Archives: 0
Standards: 75

Apache Licensed: 74
Generated Documents: 0

JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.

1 Unknown Licenses

*

Files with unapproved licenses:

  
/Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css

*





*
Summary
---
Generated at: 2017-12-08T16:33:27-05:00

Notes: 3
Binaries: 193
Archives: 0
Standards: 75

Apache Licensed: 74
Generated Documents: 0

JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.

1 Unknown Licenses

*

Files with unapproved licenses:



/Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css

*



On December 8, 2017 at 04:34:24, Matt Foley (ma...@apache.org) wrote:

Colleagues,
I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/

Given the complexity of this RC, I’d appreciate if a couple people would be
willing to kick the tires before we put it up for a vote.

I will myself be going thru the Verify Build process this weekend, as I
won’t be able to do it Friday.

Thanks,
--Matt


On 12/4/17, 2:05 PM, "zeo...@gmail.com"  wrote:

Can we resolve the conversation regarding the second repo? I was waiting
to get more input/preferences from people There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I just
need to merge it.

Jon

On Mon, Dec 4, 2017, 17:01 Casey Stella  wrote:

> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley  wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before
0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a release
> of
> > the current state of the metron-bro-plugin-kafka. I will continue the
> > discussion in that thread as to the process for accomplishing that, but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley"  wrote:
> >
> > Hope everyone (at least in the U.S.) had a great Thanksgiving
> holiday.
> > Regarding status of the release effort, still pending METRON-1252, so
> > not making the release branch yet.
> >
> > Regards,
> > --Matt
> >
> > On 11/17/17, 1:32 PM, "Matt Foley"  wrote:
> >
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near
> future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done. At this
> > point, only METRON-1252, and the discussion of how to handle joint
> release
> > of the Metron bro plugin, remain as gating items for the release. I
> > project these will be resolved next week, so let’s propose the
following:
> >
> > Sometime next week, after the last bits are done, I’ll start the
> > release process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward

Re: [DISCUSS] Upcoming Release

2017-12-08 Thread Otto Fowler

Steps

   - Downloaded tar.gz’s, asc files and KEYS
   - Verified signing of both tar.gz’s
   - searched for rouge 0.4.1 entries
   - verified the main pom.xml
   - built :

mvn clean && time mvn -q -T 2C -DskipTests install && time mvn -q -T
2C surefire:test@unit-tests && time mvn -q
surefire:test@integration-tests  && time mvn -q test --projects
metron-interface/metron-config && time build_utils/verify_licenses.sh

Found rat error:


*
Summary
---
Generated at: 2017-12-08T16:33:27-05:00

Notes: 3
Binaries: 193
Archives: 0
Standards: 75

Apache Licensed: 74
Generated Documents: 0

JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.

1 Unknown Licenses

*

Files with unapproved licenses:

  
/Users/batman/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css

*





*
Summary
---
Generated at: 2017-12-08T16:33:27-05:00

Notes: 3
Binaries: 193
Archives: 0
Standards: 75

Apache Licensed: 74
Generated Documents: 0

JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.

1 Unknown Licenses

*

Files with unapproved licenses:


/Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css

*



On December 8, 2017 at 04:34:24, Matt Foley (ma...@apache.org) wrote:

Colleagues,
I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/

Given the complexity of this RC, I’d appreciate if a couple people would be
willing to kick the tires before we put it up for a vote.

I will myself be going thru the Verify Build process this weekend, as I
won’t be able to do it Friday.

Thanks,
--Matt


On 12/4/17, 2:05 PM, "zeo...@gmail.com"  wrote:

Can we resolve the conversation regarding the second repo? I was waiting
to get more input/preferences from people There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I just
need to merge it.

Jon

On Mon, Dec 4, 2017, 17:01 Casey Stella  wrote:

> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley  wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend. Shall I go ahead and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before
0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a release
> of
> > the current state of the metron-bro-plugin-kafka. I will continue the
> > discussion in that thread as to the process for accomplishing that, but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley"  wrote:
> >
> > Hope everyone (at least in the U.S.) had a great Thanksgiving
> holiday.
> > Regarding status of the release effort, still pending METRON-1252, so
> > not making the release branch yet.
> >
> > Regards,
> > --Matt
> >
> > On 11/17/17, 1:32 PM, "Matt Foley"  wrote:
> >
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near
> future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done. At this
> > point, only METRON-1252, and the discussion of how to handle joint
> release
> > of the Metron bro plugin, remain as gating items for the release. I
> > project these will be resolved next week, so let’s propose the
following:
> >
> > Sometime next week, after the last bits are done, I’ll start the
> > release process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled
> > ‘backward-incompatible’, nor having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> > introduce backward incompatibility, please say so now on this thread,
and
> > mark the Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1
> > are listed below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > Filters Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in facet
> > and group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a
> > Kerberized

Re: [DISCUSS] Upcoming Release

2017-12-08 Thread zeo...@gmail.com

I poked around the RC briefly and spun up full dev with and without
sensors, no issues so far.

Jon

On Fri, Dec 8, 2017 at 4:34 AM Matt Foley  wrote:

> Colleagues,
> I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
>
> Given the complexity of this RC, I’d appreciate if a couple people would
> be willing to kick the tires before we put it up for a vote.
>
> I will myself be going thru the Verify Build process this weekend, as I
> won’t be able to do it Friday.
>
> Thanks,
> --Matt
>
>
> On 12/4/17, 2:05 PM, "zeo...@gmail.com"  wrote:
>
> Can we resolve the conversation regarding the second repo?  I was
> waiting
> to get more input/preferences from people  There's also a documentation
> update that fixes a few broken Stellar docs that already has aa +1, I
> just
> need to merge it.
>
> Jon
>
> On Mon, Dec 4, 2017, 17:01 Casey Stella  wrote:
>
> > I would be in favor of a release at this point.
> >
> > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley  wrote:
> >
> > > Hey all,
> > > I see METRON-1252 was resolved over the weekend.  Shall I go ahead
> and
> > > start the process with 0.4.2 release?
> > > Does anyone have any commits they feel strongly should go in
> before 0.4.2
> > > is done, or are we ready to call it good?
> > >
> > > I believe there is consensus the 0.4.2 release should include a
> release
> > of
> > > the current state of the metron-bro-plugin-kafka.  I will continue
> the
> > > discussion in that thread as to the process for accomplishing
> that, but
> > > plan on it happening.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/26/17, 6:26 PM, "Matt Foley"  wrote:
> > >
> > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > holiday.
> > > Regarding status of the release effort, still pending
> METRON-1252, so
> > > not making the release branch yet.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/17/17, 1:32 PM, "Matt Foley"  wrote:
> > >
> > > (With release manager hat on)
> > >
> > > The community has proposed a release of Metron in the near
> > future,
> > > focusing on Meta-alerts running in Elasticsearch.
> > > Congrats on getting so many of the below already done.  At
> this
> > > point, only METRON-1252, and the discussion of how to handle joint
> > release
> > > of the Metron bro plugin, remain as gating items for the release.
> I
> > > project these will be resolved next week, so let’s propose the
> following:
> > >
> > > Sometime next week, after the last bits are done, I’ll
> start the
> > > release process and create the release branch.
> > >
> > > The proposed new version will be 0.4.2, unless there are
> backward
> > > incompatible changes that support making it 0.5.0.
> > > Currently there are NO included Jiras labeled
> > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > ***If anyone knows that some of the commits included since
> 0.4.1
> > > introduce backward incompatibility, please say so now on this
> thread, and
> > > mark the Jira as such.***
> > >
> > > The 90 or so jiras/commits already in master branch since
> 0.4.1
> > > are listed below.
> > > Thanks,
> > > --Matt
> > >
> > > METRON-1301 Alerts UI - Sorting on Triage Score
> Unexpectedly
> > > Filters Some Records (nickwallen) closes apache/metron#832
> > > METRON-1294 IP addresses are not formatted correctly
> in facet
> > > and group results (merrimanr) closes apache/metron#827
> > > METRON-1291 Kafka produce REST endpoint does not work
> in a
> > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > METRON-1290 Only first 10 alerts are update when a
> MetaAlert
> > > status is changed to inactive (justinleet) closes apache/metron#842
> > > METRON-1311 Service Check Should Check Elasticsearch
> Index
> > > Templates (nickwallen) closes apache/metron#839
> > > METRON-1289 Alert fields are lost when a MetaAlert is
> created
> > > (merrimanr) closes apache/metron#824
> > > METRON-1309 Change metron-deployment to pull the
> plugin from
> > > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > > METRON-1310 Template Delete Action Deletes Search
> Indices
> > > (nickwallen) closes apache/metron#838
> > > METRON-1275: Fix Metron Documentation closes
> > > apache/incubator-metron#833
> > > METRON-1295 Unable to Configure

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

2017-12-08 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/861
  
I ran the following test:
Modified the default snort parser configuration such that it was :

```json
{
  "parserClassName":"org.apache.metron.parsers.snort.BasicSnortParser",
  "sensorTopic":"snort",
  "parserConfig": {},
  "fieldTransformations" : [
{
  "output" : ["msg" ],
  "transformation" : "SELECT"
}
  ]
}

```

And the default snort enrichment configuration such that it was :

```json

{
  "enrichment" : {
  },
  "threatIntel" : {
}
  }
}

```

I got the following:

```

2.168.138.158,49189,62.75.195.236,80,00:00:00:00:00:00,00:00:00:00:00:00,0x3C,***A,0x9DFB1927,0xF1BD72CC,,0xFAF0,128,0,2360,40,40960","enrichmentsplitterbolt.splitter.end.ts":"1512763453749","enrichmentsplitterbolt.splitter.begin.ts":"1512763453749","guid":"08a84757-bf05-431b-9d81-5fa95fb99938","timestamp":1512763452000}
at 
org.apache.metron.enrichment.bolt.EnrichmentJoinBolt.getStreamIds(EnrichmentJoinBolt.java:53)
 ~[stormjar.jar:?]
at 
org.apache.metron.enrichment.bolt.EnrichmentJoinBolt.getStreamIds(EnrichmentJoinBolt.java:33)
 ~[stormjar.jar:?]
at 
org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:138) 
[stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734)
 [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
 [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
 [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
 [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
 [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
 [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at 
org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
 [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) 
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
2017-12-08 20:04:17.171 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Trying to 
retrieve a field map with sensor type of null
2017-12-08 20:04:17.171 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Trying to 
retrieve a field map with sensor type of null
2017-12-08 20:04:17.171 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Trying to 
retrieve a field map with sensor type of null
2017-12-08 20:04:17.171 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Trying to 
retrieve a field map with sensor type of null
2017-12-08 20:04:17.171 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Trying to 
retrieve a field map with sensor type of null
2017-12-08 20:04:17.171 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Trying to 
retrieve a field map with sensor type of null
2017-12-08 20:04:17.171 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Trying to 
retrieve a field map with sensor type of null
2017-12-08 20:04:17.171 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Trying to 
retrieve a field map with sensor type of null
2017-12-08 20:04:17.171 o.a.m.e.b.EnrichmentSplitterBolt [ERROR] Trying to 
retrieve a field map with sensor type of null
```

So it looks like there are more fields to protect.



---

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

2017-12-08 Thread simonellistonball

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/861
  
Good catch, my test included the source.type in the select list, and 
generalised badly in the instructions. We should include the source.type in the 
protected system fields, let me update the test and implementation.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

2017-12-08 Thread merrimanr

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
I created a feature branch called "feature/METRON-1344-test-infrastructure" 
and switched the base of this PR to that instead of master.  What is the next 
step?


---

Re: Anyone else having problems building rpms?

2017-12-08 Thread Otto Fowler

I seem to be beyond the issue now.  But the question I have is this… do we
required internet connectivity to build rpms now?


On December 8, 2017 at 07:14:41, Otto Fowler (ottobackwa...@gmail.com)
wrote:

I’m getting errors building rpms, but not building the rest of the product
( so vagrant up makes it to the rpm phase ).
When I just build the rpms from the cli I get:

+ npm install
--prefix=/root/BUILDROOT/metron-0.4.2-root/usr/metron/0.4.2/web/expressjs
--only=production
npm ERR! Linux 4.9.49-moby
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install"
"--prefix=/root/BUILDROOT/metron-0.4.2-root/usr/metron/0.4.2/web/expressjs"
"--only=production"
npm ERR! node v6.11.3
npm ERR! npm  v3.10.10
npm ERR! code ECONNRESET

npm ERR! network tunneling socket could not be established, cause=connect
EINVAL 0.0.246.98:80 - Local (0.0.0.0:0)
npm ERR! network This is most likely not a problem with npm itself
npm ERR! network and is related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network
settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

npm ERR! Please include the following file with any support request:
npm ERR! /root/BUILD/npm-debug.log

error: Bad exit status from /var/tmp/rpm-tmp.lAmezg (%install)


Anyone else see this?

O

[GitHub] metron issue #859: METRON-1345: Update EC2 README for custom Ansible tags

2017-12-08 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/859
  
Good catch @mmiklavc.  Would be nice to get that added to the blueprint so 
it just happens.


---

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

2017-12-08 Thread simonellistonball

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/861
  
Actually, to follow up on that... I have a proxy feed, and some proxy use 
cases (enrichment, profile, etc). I want to keep my data clean and be explicit 
about which fields I pass on, so I have a 'library' field set that means "proxy 
like stuff", these are the fields I push into the output here. 


---

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

2017-12-08 Thread simonellistonball

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/861
  
If they want to select most, and remove the ones they don't want, then I 
would recommend using the remove transformation, or a set null in stellar. 
Perhaps regex support might be a nice follow on, but it breaks the mental model 
of people who are used to any other language that handles projection, such as 
SQL. The goal for this was to allow user to explicitly select only a defined 
set of fields. To be honest, people have lived with the idea of explicitly 
choosing fields for decades in SQL and quite liked it, so I suspect adding 
something that is pattern based might make it less usable than more.


---

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

2017-12-08 Thread ottobackwards

GitHub user ottobackwards reopened a pull request:

https://github.com/apache/metron/pull/856

METRON-1339 Stellar Shell functionality to verify stored stellar statements 

This will allow users to check their deployed statements, say after 
upgrade, when they are at rest ( and would fail on use ).
In other words, they were valid when stored, but are not now because of 
stellar changes, such as new keywords.

The interface `StellarConfiguredStatementReporter`, which is 
`@IndexSubclasses` ( ClassIndex) marked, allows the shell to discover reporters 
that can provide statements for validation.  This discovery allows de-coupling 
of stellar and 'hosts' that know about the location of the stored statements, 
and the configuration structure details.

> We do mention the configurations in the shell output at this time.

`metron-common` implements this interface, and can run through visiting all 
the configurations.

A new magic keyword was added ` %validate_configured_expressions`
When executed, the shell 

- discovers the reporters through class index 
- visits the reports, with callbacks for visits or errors
- per visit ( which is called for a specific stellar statement ) the 
statement is compiled and errors reported
- if the entire config fails ( threat triage stellar errors fail on 
deserialize so we don't get to do ANY enrichment visits in that case ) the 
error callback handles that

I'm getting this out there, still a couple of things todo:

[x] ~~full dev run. I have been testing with stellar external to full dev 
iteratively~~
[x] ~~readme~~
[x] ~~steps to test~~
[x] ~~unit test~~
[x] ~~ThreatTriage Rule Reason~~


## Testing
- deploy full dev
- edit the squid parser transformation(s) such that the stellar would not 
compile, such as adding a dangling  `=` in zookeeper
```json
{ 
"parserClassName": "org.apache.metron.parsers.GrokParser", 
"sensorTopic": "squid", 
"parserConfig": { 
"grokPath": "/patterns/squid", 
"patternLabel": "SQUID_DELIMITED", 
"timestampField": "timestamp" 
}, 
"fieldTransformations" : [ 
{ 
"transformation" : "STELLAR" 
,"output" : [ "full_hostname", "domain_without_subdomains" ] 
,"config" : { 
"full_hostname" : "URL_TO_HOST(url) =" 
,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)" 
} 
} 
] 
}

```

- edit the snort threat triage rules in it's enrichment config in zookeeper 
( here with an extra `)` )

```json
{ 
"enrichment" : { 
"fieldMap": 
{ 
"geo": ["ip_dst_addr", "ip_src_addr"], 
"host": ["host"] 
} 
}, 
"threatIntel" : { 
"fieldMap": 
{ 
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"] 
}, 
"fieldToTypeMap": 
{ 
"ip_src_addr" : ["malicious_ip"], 
"ip_dst_addr" : ["malicious_ip"] 
}, 
"triageConfig" : { 
"riskLevelRules" : [ 
{ 
"rule" : "not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24')) )", 
"score" : 10 
} 
], 
"aggregator" : "MAX" 
} 
} 
} 
```

## Working with zookeeper
I am not a zk cli maestro, so I took the easy way out and used 
[ZK-WEB](https://github.com/qiuxiafei/zk-web).
Following the readme instructions it was very simple to clone, edit the 
config for full dev, and run from source.  If you log in with the creds in the 
config you can edit the nodes.


## Results
When you run the magic command, it will report the failed stellar 
statements, and the failed enrichment config:

```bash
[Stellar]>>> %validate_configured_expressions
Discovered 1 reporters
Visiting all configurations.  ThreatTriage rules are checked when loading 
the configuration, thus an invalid ThreatTriage rule will fail the entire 
Enrichement Configuration.
Apache Metron
Visiting Apache Metron


==


validating Apache Metron->PARSER->squid->full_hostname
[!] Error Visiting Apache Metron->PARSER->squid->full_hostname
Syntax error @ 1:17 token recognition error at: '='
--
[!] : URL_TO_HOST(url) =


==




==


validating Apache Metron->PARSER->squid->domain_without_subdomains


==


[!] Configuration Apache Metron->ENRICHMENT->snort is not valid, please 
review

Done validation
[Stellar]>>>
```

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

2017-12-08 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
Closing to test build in travis


---

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

2017-12-08 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
Bump?


---

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

2017-12-08 Thread ottobackwards

Github user ottobackwards closed the pull request at:

https://github.com/apache/metron/pull/856


---

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

2017-12-08 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/861
  
I trying to run this up in full dev.  I have two issues:

1.  expressJS is failing download, obviously outside this PR
2. The usability of this transform.  If a user just wants a couple of 
fields, then no problem, but If they want anything more than that, then 
selecting all the fields that they want seems like a bit of a chore.  So if 
they want to select *most* of the fields, they have to list them all out.   It 
seems tough.  I wonder if we don't need regex support or something.  

Thoughts?


---

Anyone else having problems building rpms?

2017-12-08 Thread Otto Fowler

I’m getting errors building rpms, but not building the rest of the product
( so vagrant up makes it to the rpm phase ).
When I just build the rpms from the cli I get:

+ npm install
--prefix=/root/BUILDROOT/metron-0.4.2-root/usr/metron/0.4.2/web/expressjs
--only=production
npm ERR! Linux 4.9.49-moby
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install"
"--prefix=/root/BUILDROOT/metron-0.4.2-root/usr/metron/0.4.2/web/expressjs"
"--only=production"
npm ERR! node v6.11.3
npm ERR! npm  v3.10.10
npm ERR! code ECONNRESET

npm ERR! network tunneling socket could not be established, cause=connect
EINVAL 0.0.246.98:80 - Local (0.0.0.0:0)
npm ERR! network This is most likely not a problem with npm itself
npm ERR! network and is related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network
settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

npm ERR! Please include the following file with any support request:
npm ERR! /root/BUILD/npm-debug.log

error: Bad exit status from /var/tmp/rpm-tmp.lAmezg (%install)


Anyone else see this?

O

Re: [DISCUSS] Upcoming Release

2017-12-08 Thread Matt Foley

Colleagues,
I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to 
https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/

Given the complexity of this RC, I’d appreciate if a couple people would be 
willing to kick the tires before we put it up for a vote.

I will myself be going thru the Verify Build process this weekend, as I won’t 
be able to do it Friday.

Thanks,
--Matt


On 12/4/17, 2:05 PM, "zeo...@gmail.com"  wrote:

Can we resolve the conversation regarding the second repo?  I was waiting
to get more input/preferences from people  There's also a documentation
update that fixes a few broken Stellar docs that already has aa +1, I just
need to merge it.

Jon

On Mon, Dec 4, 2017, 17:01 Casey Stella  wrote:

> I would be in favor of a release at this point.
>
> On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley  wrote:
>
> > Hey all,
> > I see METRON-1252 was resolved over the weekend.  Shall I go ahead and
> > start the process with 0.4.2 release?
> > Does anyone have any commits they feel strongly should go in before 
0.4.2
> > is done, or are we ready to call it good?
> >
> > I believe there is consensus the 0.4.2 release should include a release
> of
> > the current state of the metron-bro-plugin-kafka.  I will continue the
> > discussion in that thread as to the process for accomplishing that, but
> > plan on it happening.
> >
> > Regards,
> > --Matt
> >
> > On 11/26/17, 6:26 PM, "Matt Foley"  wrote:
> >
> > Hope everyone (at least in the U.S.) had a great Thanksgiving
> holiday.
> > Regarding status of the release effort, still pending METRON-1252, 
so
> > not making the release branch yet.
> >
> > Regards,
> > --Matt
> >
> > On 11/17/17, 1:32 PM, "Matt Foley"  wrote:
> >
> > (With release manager hat on)
> >
> > The community has proposed a release of Metron in the near
> future,
> > focusing on Meta-alerts running in Elasticsearch.
> > Congrats on getting so many of the below already done.  At this
> > point, only METRON-1252, and the discussion of how to handle joint
> release
> > of the Metron bro plugin, remain as gating items for the release.  I
> > project these will be resolved next week, so let’s propose the 
following:
> >
> > Sometime next week, after the last bits are done, I’ll start the
> > release process and create the release branch.
> >
> > The proposed new version will be 0.4.2, unless there are 
backward
> > incompatible changes that support making it 0.5.0.
> > Currently there are NO included Jiras labeled
> > ‘backward-incompatible’, nor having Docs Text indicating so.
> > ***If anyone knows that some of the commits included since 0.4.1
> > introduce backward incompatibility, please say so now on this thread, 
and
> > mark the Jira as such.***
> >
> > The 90 or so jiras/commits already in master branch since 0.4.1
> > are listed below.
> > Thanks,
> > --Matt
> >
> > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > Filters Some Records (nickwallen) closes apache/metron#832
> > METRON-1294 IP addresses are not formatted correctly in 
facet
> > and group results (merrimanr) closes apache/metron#827
> > METRON-1291 Kafka produce REST endpoint does not work in a
> > Kerberized cluster (merrimanr) closes apache/metron#826
> > METRON-1290 Only first 10 alerts are update when a MetaAlert
> > status is changed to inactive (justinleet) closes apache/metron#842
> > METRON-1311 Service Check Should Check Elasticsearch Index
> > Templates (nickwallen) closes apache/metron#839
> > METRON-1289 Alert fields are lost when a MetaAlert is 
created
> > (merrimanr) closes apache/metron#824
> > METRON-1309 Change metron-deployment to pull the plugin from
> > apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
> > METRON-1310 Template Delete Action Deletes Search Indices
> > (nickwallen) closes apache/metron#838
> > METRON-1275: Fix Metron Documentation closes
> > apache/incubator-metron#833
> > METRON-1295 Unable to Configure Logging for REST API
> > (nickwallen) closes apache/metron#828
> > METRON-1307 Force install of java8 since java9 does not
> appear
> > to work with the scripts (brianhurley via ottobackwards) closes
> > apache/metron#835
> > METRON-1296 Full Dev Fails to Deploy Index Templates
> > (nickwallen via cestella) closes apache/incubator-metron#829

Re: [DISCUSS] Upcoming Release

Re: [DISCUSS] Upcoming Release

Re: [DISCUSS] Upcoming Release

Re: [DISCUSS] Upcoming Release

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Re: Anyone else having problems building rpms?

[GitHub] metron issue #859: METRON-1345: Update EC2 README for custom Ansible tags

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

Anyone else having problems building rpms?

Re: [DISCUSS] Upcoming Release

18 matches

Site Navigation

Mail list logo

Footer information