[GitHub] incubator-metron pull request: METRON-103: Unit Tests for storm bo...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/81#discussion_r60152962
  
--- Diff: deployment/roles/metron_ui/tasks/main.yml ---
@@ -53,4 +53,4 @@
 production: no
 
 - name: Start Metron UI
-  shell: "pm2 restart {{ metron_ui_directory }}/lib/metron-ui.js --name 
metron"
+  shell: "pm2 start {{ metron_ui_directory }}/lib/metron-ui.js --name 
metron"
--- End diff --

Oh, good call then. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request: METRON-103: Unit Tests for storm bo...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/81#discussion_r60152303
  
--- Diff: deployment/roles/metron_ui/tasks/main.yml ---
@@ -53,4 +53,4 @@
 production: no
 
 - name: Start Metron UI
-  shell: "pm2 restart {{ metron_ui_directory }}/lib/metron-ui.js --name 
metron"
+  shell: "pm2 start {{ metron_ui_directory }}/lib/metron-ui.js --name 
metron"
--- End diff --

Yes this was intentional.  Turns out the previous commit only works on AWS 
so I reverted it to keep the single node vagrant deployment from breaking.  
@nickwallen reopened the pull request and will submit another fix.  Sorry I 
should have called that out in the description.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request: METRON-103: Unit Tests for storm bo...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/81#discussion_r60145331
  
--- Diff: deployment/roles/metron_ui/tasks/main.yml ---
@@ -53,4 +53,4 @@
 production: no
 
 - name: Start Metron UI
-  shell: "pm2 restart {{ metron_ui_directory }}/lib/metron-ui.js --name 
metron"
+  shell: "pm2 start {{ metron_ui_directory }}/lib/metron-ui.js --name 
metron"
--- End diff --

Did you intend to revert this change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Project reorganization

[James Sirota]

Hi Ryan,

This is great.  You should attach this to the Jira when you are ready to commit 
the reorg so we know which parts shifted.

Thanks,
James 




On 4/18/16, 1:30 PM, "Ryan Merriman"  wrote:

>Thanks Frank.  I’ve updated those in the spreadsheet.
>
>On 4/18/16, 3:27 PM, "Frank Lu"  wrote:
>
>>As of now, I think the following classes are not used:
>>
>>
>> 
>> 
>>Metron-EnrichmentAdapters
>>  org.apache.metron.enrichment.adapters.cif.AbstractCIFAdapter.java
>> 
>> 
>>  org.apache.metron.enrichment.adapters.cif.CIFHbaseAdapter.java
>>
>>org.apache.metron.enrichment.adapters.whois.WhoisHBaseAdapter.java
>>
>>
>>Metron-DataLoads
>>org.apache.metron.dataloads.cif.HBaseTableLoad.java
>>  
>>
>>Thanks,
>>Frank Lu
>>
>>
>>
>>
>>On 4/18/16, 3:05 PM, "Ryan Merriman"  wrote:
>>
>>>All,
>>>
>>>I put together a list of all the project java assets that details where
>>>they will be moved (or potentially deleted) as part of the project
>>>reorganization.  Feedback welcome.
>>>
>>>Ryan Merriman 
>>>
>>>On 4/13/16, 9:42 AM, "James Sirota"  wrote:
>>>
I would have configs as a project but rather as a folder structure that
other modules can point to

Thanks,
James 




On 4/13/16, 7:32 AM, "Ryan Merriman"  wrote:

>James brings up a good point.  I propose adding another project under
>metron-platform called metron-configuration.  This would be a fairly
>lightweight project that would contain anything related to
>configuration
>(property files, json files, flux files, etc).
>
>On 4/13/16, 8:56 AM, "James Sirota"  wrote:
>
>>+1 from me.
>>
>>I would also like to address the configs and make sure the configs are
>>in
>>the same place.  Do you have ideas on where we would put those?
>>
>>Thanks,
>>James 
>>
>>
>>
>>On 4/13/16, 6:50 AM, "Ryan Merriman" 
>>wrote:
>>
>>>Thank you for all the feedback everyone.  I will attempt to summarize
>>>all
>>>the input we¹ve received and update my initial proposal.  We can
>>>discuss
>>>further if anyone is still unclear and I will volunteer to capture
>>>all
>>>the
>>>details in a document of some kind once we all come to a consensus.
>>>
>>>Looks like everyone is in agreement for the top level projects.  Nick
>>>is
>>>working on a task that will require an addition top level project so
>>>I
>>>am
>>>going to add that in as well:
>>>
>>>metron-deployment
>>>metron-platform
>>>metron-ui
>>>metron-sensors
>>>
>>>All of these except metron-platform are well understood and don¹t
>>>warrant
>>>any more discussion.  For metron-platform there seem to be 2 areas
>>>that
>>>are not as clear:
>>>
>>>- whether we need a common project
>>>- how do we organize test related code
>>>
>>>I agree with David and others that a common project will likely get
>>>misused and could become unnecessary bloated.  But I suspect there
>>>will
>>>be
>>>cases where we have common code being used across multiple projects
>>>(is
>>>already happening).  In this case we will either need this common
>>>project
>>>or we will have to keep common code in one of the other projects and
>>>have
>>>all other projects extend that. For the latter, an example would be
>>>keeping common code in enrichment and having parsers declare
>>>enrichment
>>>as
>>>a dependency.  There are a couple downsides I see with this approach:
>>>
>>>- parser topology jars now bring along all the enrichment
>>>dependencies
>>>- since more code from various projects are being packaged together,
>>>version conflicts are more likely and poms become more complicated
>>>due
>>>to
>>>all the necessary exclusions
>>>
>>>My thinking is that any jar file being deployed should only contain
>>>what
>>>it needs.  Curious what others think here.  My vote would be to
>>>maintain
>>>a
>>>common project (or whatever we want to call it) and be diligent about
>>>not
>>>letting project-specific code slip in there.
>>>
>>>I believe Nick was the first person to ask the question about
>>>projects
>>>related to test code and why we would need separate test and
>>>integration
>>>test.  The reason for this is that our integration-test classes
>>>currently
>>>depend on other projects (not surprising since they are integration
>>>tests).  If there are utilities we want make available to all
>>>projects
>>>(mock classes, utilities for reading sample data, etc) then it can¹t
>>>live
>>>in integration-test because that will introduce circular
>>>dependencies.

Re: [DISCUSS] Project reorganization

[Ryan Merriman]

Thanks Frank.  I’ve updated those in the spreadsheet.

On 4/18/16, 3:27 PM, "Frank Lu"  wrote:

>As of now, I think the following classes are not used:
>
>
> 
> 
>Metron-EnrichmentAdapters
>  org.apache.metron.enrichment.adapters.cif.AbstractCIFAdapter.java
> 
> 
>  org.apache.metron.enrichment.adapters.cif.CIFHbaseAdapter.java
>
>org.apache.metron.enrichment.adapters.whois.WhoisHBaseAdapter.java
>
>
>Metron-DataLoads
>org.apache.metron.dataloads.cif.HBaseTableLoad.java
>   
>
>Thanks,
>Frank Lu
>
>
>
>
>On 4/18/16, 3:05 PM, "Ryan Merriman"  wrote:
>
>>All,
>>
>>I put together a list of all the project java assets that details where
>>they will be moved (or potentially deleted) as part of the project
>>reorganization.  Feedback welcome.
>>
>>Ryan Merriman 
>>
>>On 4/13/16, 9:42 AM, "James Sirota"  wrote:
>>
>>>I would have configs as a project but rather as a folder structure that
>>>other modules can point to
>>>
>>>Thanks,
>>>James 
>>>
>>>
>>>
>>>
>>>On 4/13/16, 7:32 AM, "Ryan Merriman"  wrote:
>>>
James brings up a good point.  I propose adding another project under
metron-platform called metron-configuration.  This would be a fairly
lightweight project that would contain anything related to
configuration
(property files, json files, flux files, etc).

On 4/13/16, 8:56 AM, "James Sirota"  wrote:

>+1 from me.
>
>I would also like to address the configs and make sure the configs are
>in
>the same place.  Do you have ideas on where we would put those?
>
>Thanks,
>James 
>
>
>
>On 4/13/16, 6:50 AM, "Ryan Merriman" 
>wrote:
>
>>Thank you for all the feedback everyone.  I will attempt to summarize
>>all
>>the input we¹ve received and update my initial proposal.  We can
>>discuss
>>further if anyone is still unclear and I will volunteer to capture
>>all
>>the
>>details in a document of some kind once we all come to a consensus.
>>
>>Looks like everyone is in agreement for the top level projects.  Nick
>>is
>>working on a task that will require an addition top level project so
>>I
>>am
>>going to add that in as well:
>>
>>metron-deployment
>>metron-platform
>>metron-ui
>>metron-sensors
>>
>>All of these except metron-platform are well understood and don¹t
>>warrant
>>any more discussion.  For metron-platform there seem to be 2 areas
>>that
>>are not as clear:
>>
>>- whether we need a common project
>>- how do we organize test related code
>>
>>I agree with David and others that a common project will likely get
>>misused and could become unnecessary bloated.  But I suspect there
>>will
>>be
>>cases where we have common code being used across multiple projects
>>(is
>>already happening).  In this case we will either need this common
>>project
>>or we will have to keep common code in one of the other projects and
>>have
>>all other projects extend that. For the latter, an example would be
>>keeping common code in enrichment and having parsers declare
>>enrichment
>>as
>>a dependency.  There are a couple downsides I see with this approach:
>>
>>- parser topology jars now bring along all the enrichment
>>dependencies
>>- since more code from various projects are being packaged together,
>>version conflicts are more likely and poms become more complicated
>>due
>>to
>>all the necessary exclusions
>>
>>My thinking is that any jar file being deployed should only contain
>>what
>>it needs.  Curious what others think here.  My vote would be to
>>maintain
>>a
>>common project (or whatever we want to call it) and be diligent about
>>not
>>letting project-specific code slip in there.
>>
>>I believe Nick was the first person to ask the question about
>>projects
>>related to test code and why we would need separate test and
>>integration
>>test.  The reason for this is that our integration-test classes
>>currently
>>depend on other projects (not surprising since they are integration
>>tests).  If there are utilities we want make available to all
>>projects
>>(mock classes, utilities for reading sample data, etc) then it can¹t
>>live
>>in integration-test because that will introduce circular
>>dependencies.
>>If
>>it is possible to refactor our current Metron-Testing project so that
>>it
>>doesn¹t depend on any other projects, then we can keep utilities
>>here.
>>Otherwise we need a separate project for testing utilities.  I
>>suspect
>>removing other project dependencies from Metron-Testing will prove
>>more
>>difficult than it¹s worth 

Re: [DISCUSS] Metron Logo Options

[James Sirota]

#1 does look like Metaspoit, but I think that’s the right direction.  It’s sort 
of security(ish).  I’d like to dig into that some more 




On 4/18/16, 12:22 PM, "zeo...@gmail.com"  wrote:

>The first looks a little to close to Metasploit for my comfort.  I have no
>opinion on the others - they look good though.
>
>Jon
>
>On Mon, Apr 18, 2016 at 3:13 PM Houshang Livian 
>wrote:
>
>> Apologies for the flood of emails. Here is a dropbox link for those that
>> could not see the options:
>> https://www.dropbox.com/s/6hxz76aet8gzt0a/metron-logo-options.png?dl=0
>>
>>
>> From: default >
>> Reply-To: "dev@metron.incubator.apache.org> dev@metron.incubator.apache.org>"  dev@metron.incubator.apache.org>>
>> Date: Monday, April 18, 2016 at 12:07 PM
>> To: "dev@metron.incubator.apache.org> dev@metron.incubator.apache.org>"  dev@metron.incubator.apache.org>>
>> Subject: Re: [DISCUSS] Metron Logo Options
>>
>> Trying again with the options as an attachment.
>>
>> From: default >
>> Reply-To: "dev@metron.incubator.apache.org> dev@metron.incubator.apache.org>"  dev@metron.incubator.apache.org>>
>> Date: Monday, April 18, 2016 at 12:05 PM
>> To: "dev@metron.incubator.apache.org> dev@metron.incubator.apache.org>"  dev@metron.incubator.apache.org>>
>> Subject: [DISCUSS] Metron Logo Options
>>
>> Hey Guys,
>>
>> As you know, Metron is based on a DC Comic Book character.
>>
>> Here are 5 different logo options for Metron. Let me know your thoughts
>> and your votes.
>>  [cid:FC8299C6-AAAB-4CC2-9129-3C19479F1D8F]
>>
>> Houshang Livian
>> Senior User Experience Designer
>> Hortonworks
>>
>> www.hortonworks.com
>> 
>>
>> Mobile: (831) 521-4176
>> hliv...@hortonworks.com
>> 
>>
>> --
>
>Jon

Re: [DISCUSS] Metron Logo Options

[Houshang Livian]

Hi Charles,

If you can find the logo you liked I can add it to the options.




On 4/18/16, 12:56 PM, "charles.por...@gmail.com on behalf of Charles Porter" 
 wrote:

>I thought we had this vote a couple of months ago. I liked some (at least
>one) of those old options.
>
>On Mon, Apr 18, 2016 at 12:22 PM, zeo...@gmail.com  wrote:
>
>> The first looks a little to close to Metasploit for my comfort.  I have no
>> opinion on the others - they look good though.
>>
>> Jon
>>
>> On Mon, Apr 18, 2016 at 3:13 PM Houshang Livian 
>> wrote:
>>
>> > Apologies for the flood of emails. Here is a dropbox link for those that
>> > could not see the options:
>> > https://www.dropbox.com/s/6hxz76aet8gzt0a/metron-logo-options.png?dl=0
>> >
>> >
>> > From: default >
>> > Reply-To: "dev@metron.incubator.apache.org> > dev@metron.incubator.apache.org>" > > > dev@metron.incubator.apache.org>>
>> > Date: Monday, April 18, 2016 at 12:07 PM
>> > To: "dev@metron.incubator.apache.org> > dev@metron.incubator.apache.org>" > > > dev@metron.incubator.apache.org>>
>> > Subject: Re: [DISCUSS] Metron Logo Options
>> >
>> > Trying again with the options as an attachment.
>> >
>> > From: default >
>> > Reply-To: "dev@metron.incubator.apache.org> > dev@metron.incubator.apache.org>" > > > dev@metron.incubator.apache.org>>
>> > Date: Monday, April 18, 2016 at 12:05 PM
>> > To: "dev@metron.incubator.apache.org> > dev@metron.incubator.apache.org>" > > > dev@metron.incubator.apache.org>>
>> > Subject: [DISCUSS] Metron Logo Options
>> >
>> > Hey Guys,
>> >
>> > As you know, Metron is based on a DC Comic Book character.
>> >
>> > Here are 5 different logo options for Metron. Let me know your thoughts
>> > and your votes.
>> >  [cid:FC8299C6-AAAB-4CC2-9129-3C19479F1D8F]
>> >
>> > Houshang Livian
>> > Senior User Experience Designer
>> > Hortonworks
>> >
>> > www.hortonworks.com
>> > 
>> >
>> > Mobile: (831) 521-4176
>> > hliv...@hortonworks.com
>> > 
>> >
>> > --
>>
>> Jon
>>

Re: [DISCUSS] Project reorganization

[Ryan Merriman]

All,

I put together a list of all the project java assets that details where
they will be moved (or potentially deleted) as part of the project
reorganization.  Feedback welcome.

Ryan Merriman 

On 4/13/16, 9:42 AM, "James Sirota"  wrote:

>I would have configs as a project but rather as a folder structure that
>other modules can point to
>
>Thanks,
>James 
>
>
>
>
>On 4/13/16, 7:32 AM, "Ryan Merriman"  wrote:
>
>>James brings up a good point.  I propose adding another project under
>>metron-platform called metron-configuration.  This would be a fairly
>>lightweight project that would contain anything related to configuration
>>(property files, json files, flux files, etc).
>>
>>On 4/13/16, 8:56 AM, "James Sirota"  wrote:
>>
>>>+1 from me.
>>>
>>>I would also like to address the configs and make sure the configs are
>>>in
>>>the same place.  Do you have ideas on where we would put those?
>>>
>>>Thanks,
>>>James 
>>>
>>>
>>>
>>>On 4/13/16, 6:50 AM, "Ryan Merriman"  wrote:
>>>
Thank you for all the feedback everyone.  I will attempt to summarize
all
the input we¹ve received and update my initial proposal.  We can
discuss
further if anyone is still unclear and I will volunteer to capture all
the
details in a document of some kind once we all come to a consensus.

Looks like everyone is in agreement for the top level projects.  Nick
is
working on a task that will require an addition top level project so I
am
going to add that in as well:

metron-deployment
metron-platform
metron-ui
metron-sensors

All of these except metron-platform are well understood and don¹t
warrant
any more discussion.  For metron-platform there seem to be 2 areas that
are not as clear:

- whether we need a common project
- how do we organize test related code

I agree with David and others that a common project will likely get
misused and could become unnecessary bloated.  But I suspect there will
be
cases where we have common code being used across multiple projects (is
already happening).  In this case we will either need this common
project
or we will have to keep common code in one of the other projects and
have
all other projects extend that. For the latter, an example would be
keeping common code in enrichment and having parsers declare enrichment
as
a dependency.  There are a couple downsides I see with this approach:

- parser topology jars now bring along all the enrichment dependencies
- since more code from various projects are being packaged together,
version conflicts are more likely and poms become more complicated due
to
all the necessary exclusions

My thinking is that any jar file being deployed should only contain
what
it needs.  Curious what others think here.  My vote would be to
maintain
a
common project (or whatever we want to call it) and be diligent about
not
letting project-specific code slip in there.

I believe Nick was the first person to ask the question about projects
related to test code and why we would need separate test and
integration
test.  The reason for this is that our integration-test classes
currently
depend on other projects (not surprising since they are integration
tests).  If there are utilities we want make available to all projects
(mock classes, utilities for reading sample data, etc) then it can¹t
live
in integration-test because that will introduce circular dependencies.
If
it is possible to refactor our current Metron-Testing project so that
it
doesn¹t depend on any other projects, then we can keep utilities here.
Otherwise we need a separate project for testing utilities.  I suspect
removing other project dependencies from Metron-Testing will prove more
difficult than it¹s worth so my vote would be to have 2 test related
projects.

So here is where our metron-platform organization stands:

metron-common *
metron-integration-test *
metron-test-utilities *
metron-data-management
metron-pcap
metron-parsers
metron-enrichment
metron-solr
metron-elasticsearch
metron-api

* may or may not change depending on the outcome of this discussion

Thoughts?

Ryan Merriman


On 4/11/16, 4:15 PM, "Debojyoti Dutta"  wrote:

>If you load up your Irc client just type
>/join #apache-metron-dev
>
>Sent from my iPhone
>
>> On Apr 11, 2016, at 12:06 PM, James Sirota 
>>wrote:
>> 
>> Great, thanks, Debo.  Where can I find instructions on how to get to
>>it?
>> 
>> Thanks,
>> James 
>> 
>> 
>> 
>> 
>>> On 

Re: [DISCUSS] Metron Logo Options

[Charles Porter]

I thought we had this vote a couple of months ago. I liked some (at least
one) of those old options.

On Mon, Apr 18, 2016 at 12:22 PM, zeo...@gmail.com  wrote:

> The first looks a little to close to Metasploit for my comfort.  I have no
> opinion on the others - they look good though.
>
> Jon
>
> On Mon, Apr 18, 2016 at 3:13 PM Houshang Livian 
> wrote:
>
> > Apologies for the flood of emails. Here is a dropbox link for those that
> > could not see the options:
> > https://www.dropbox.com/s/6hxz76aet8gzt0a/metron-logo-options.png?dl=0
> >
> >
> > From: default >
> > Reply-To: "dev@metron.incubator.apache.org > dev@metron.incubator.apache.org>"   > dev@metron.incubator.apache.org>>
> > Date: Monday, April 18, 2016 at 12:07 PM
> > To: "dev@metron.incubator.apache.org > dev@metron.incubator.apache.org>"   > dev@metron.incubator.apache.org>>
> > Subject: Re: [DISCUSS] Metron Logo Options
> >
> > Trying again with the options as an attachment.
> >
> > From: default >
> > Reply-To: "dev@metron.incubator.apache.org > dev@metron.incubator.apache.org>"   > dev@metron.incubator.apache.org>>
> > Date: Monday, April 18, 2016 at 12:05 PM
> > To: "dev@metron.incubator.apache.org > dev@metron.incubator.apache.org>"   > dev@metron.incubator.apache.org>>
> > Subject: [DISCUSS] Metron Logo Options
> >
> > Hey Guys,
> >
> > As you know, Metron is based on a DC Comic Book character.
> >
> > Here are 5 different logo options for Metron. Let me know your thoughts
> > and your votes.
> >  [cid:FC8299C6-AAAB-4CC2-9129-3C19479F1D8F]
> >
> > Houshang Livian
> > Senior User Experience Designer
> > Hortonworks
> >
> > www.hortonworks.com
> > 
> >
> > Mobile: (831) 521-4176
> > hliv...@hortonworks.com
> > 
> >
> > --
>
> Jon
>

Re: [DISCUSS] Metron Logo Options

[zeo...@gmail.com]

The first looks a little to close to Metasploit for my comfort.  I have no
opinion on the others - they look good though.

Jon

On Mon, Apr 18, 2016 at 3:13 PM Houshang Livian 
wrote:

> Apologies for the flood of emails. Here is a dropbox link for those that
> could not see the options:
> https://www.dropbox.com/s/6hxz76aet8gzt0a/metron-logo-options.png?dl=0
>
>
> From: default >
> Reply-To: "dev@metron.incubator.apache.org dev@metron.incubator.apache.org>" >
> Date: Monday, April 18, 2016 at 12:07 PM
> To: "dev@metron.incubator.apache.org dev@metron.incubator.apache.org>" >
> Subject: Re: [DISCUSS] Metron Logo Options
>
> Trying again with the options as an attachment.
>
> From: default >
> Reply-To: "dev@metron.incubator.apache.org dev@metron.incubator.apache.org>" >
> Date: Monday, April 18, 2016 at 12:05 PM
> To: "dev@metron.incubator.apache.org dev@metron.incubator.apache.org>" >
> Subject: [DISCUSS] Metron Logo Options
>
> Hey Guys,
>
> As you know, Metron is based on a DC Comic Book character.
>
> Here are 5 different logo options for Metron. Let me know your thoughts
> and your votes.
>  [cid:FC8299C6-AAAB-4CC2-9129-3C19479F1D8F]
>
> Houshang Livian
> Senior User Experience Designer
> Hortonworks
>
> www.hortonworks.com
> 
>
> Mobile: (831) 521-4176
> hliv...@hortonworks.com
> 
>
> --

Jon