[DISCUSS] MPack components that don't support Kerberos

There is a PR up for review ( https://github.com/apache/incubator-metron/pull/518) that updates our MPack to support a Kerberized environment. There is also a PR up for review that adds the REST service to the MPack ( https://github.com/apache/incubator-metron/pull/500). However, the REST

Re: metron UI

That PR was merged this morning. Do a pull and you should see it in master. On Tue, Apr 11, 2017 at 11:44 AM, moshe jarusalem wrote: > Hi All, > Am I missing something basic ? or something other? Would you promptly > redirect to a doc ? > > On Tue, Apr 11, 2017 at 4:08 PM,

Re: [DISCUSS] next release proposal

We just finished responding to the first round of feedback so I don't think we're that far away on METRON-623. On Wed, Apr 5, 2017 at 3:30 PM, Matt Foley wrote: > Totally agree would be good to have MPack support. Let’s see how it > goes. Wouldn’t want to cut it out for the

Re: [MENTORS] Minified javascript license

es should > be updated appropriately. > > -Taylor > > > On Apr 5, 2017, at 10:25 PM, James Sirota <jsir...@apache.org> wrote: > > > > Wanted to bump this up to the top. Did we ever get a resolution on this? > > > > 05.04.2017, 05:51, "Ryan Mer

[MENTORS] Minified javascript license

be applied? If it should be applied as a comment then why not just apply the normal header? Thanks in advance for any help or advice you can give me. I am not able to find a clear example or explanation and want to make sure this gets done correctly. Ryan Merriman

Re: [DISCUSS][PROPOSAL] Acceptance Tests

gt; Maybe, but I'd argue that we would want this to be run against a > > > non-ansible installed cluster. For a first pass, I'd recommend just a > > set > > > of shell scripts utilizing the REPL and the REST API along with shell > > > commands. Most of our capabi

Re: [DISCUSS][PROPOSAL] Acceptance Tests

I think we'll have non manual rest/web deployment soon regardless of this discussion. On Wed, Mar 22, 2017 at 2:00 PM, Ryan Merriman <merrim...@gmail.com> wrote: > I don't think a cluster installed by ansible is a prerequisite to using > ansible to integration test. They would b

Re: [DISCUSS][PROPOSAL] Acceptance Tests

nd the REST API along with shell > commands. Most of our capabilities are quite scriptable. > > On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman <merrim...@gmail.com> > wrote: > > > Bumping this thread. Looks like we have several +1s so I propose we move > > to the

Re: [DISCUSS][PROPOSAL] Acceptance Tests

Bumping this thread. Looks like we have several +1s so I propose we move to the next step. I'm anxious to get this done because these tests would have saved me time over the last couple weeks. The management UI in https://github.com/apache/incubator-metron/pull/484 has a set of e2e tests being

Re: [DISCUSS] Stepping down as release manager

+1 for Matt On Tue, Mar 21, 2017 at 9:44 AM, Matt Foley wrote: > Casey, you’ve been a great release manager. I know how much detail effort > goes into this role. > > I am willing to serve as RM for the next while, if the community would > like. I was the RM for Hadoop for

Re: [GitHub] incubator-metron issue #477: METRON-766: Release 0.3.1

The delete topic function is Kafka has had some issues in the past. I don't think it's critical we expose that through the REST API so I propose we just take it out. Any objections? On Thu, Mar 16, 2017 at 3:52 PM, cestella wrote: > Github user cestella commented on the

Re: [VOTE] Metron to graduate to TLP

+1 (binding) > On Mar 13, 2017, at 6:05 PM, Casey Stella wrote: > > +1 (binding) > >> On Mon, Mar 13, 2017 at 6:37 PM, James Sirota wrote: >> >> +1 (binding) >> >> 13.03.2017, 15:37, "James Sirota" : >>> Do we feel it's time for us

Re: [VOTE] Cesey Stella for Metron VP

+1 (binding) > On Mar 13, 2017, at 6:04 PM, Justin Leet wrote: > > +1 (non-binding) > >> On Mon, Mar 13, 2017 at 6:35 PM, zeo...@gmail.com wrote: >> >> +1 (non-binding) >> >>> On Mon, Mar 13, 2017 at 6:34 PM James Sirota wrote:

Re: metron-rest - what is the default user and password

I will get that added to the README. On Wed, Mar 8, 2017 at 1:22 PM, Otto Fowler wrote: > Never mind. found it in the tests. > > > > On March 8, 2017 at 14:18:22, Otto Fowler (ottobackwa...@gmail.com) wrote: > > I’m trying to run the metron-interface/metron-rest

Re: Metron Rest - where is reflections coming from?

I have a good understanding of this since I spent a good amount of time troubleshooting it. Let me attempt to explain it. Hopefully if everyone clearly understands the core issue we can put our heads together for a solution. The metron-common module shades guava and is installed as an uber jar

Re: Metron Rest - where is reflections coming from?

Here is an explanation: https://github.com/apache/incubator-metron/pull/316#issuecomment-282791185 What exactly do you mean by "build is failing". Are you trying to spin it up in Intellij? If so you'll need to add the metron-common jar that is installed in your local Maven repo (the uber jar

Re: [DISCUSS][PROPOSAL] Acceptance Tests

+1, great idea. At some point our manual testing checklist is going to grow large enough that we'll need to move to something more automated. We're probably already there. I very much agree with Justin's concern. Building and running unit/integration tests takes a long time right now and this

Re: Travis CI Changes have broken my heart..... or my builds

I run into this error every now and then. If I free up space on my hard drive it goes away. > On Feb 27, 2017, at 8:31 PM, Otto Fowler wrote: > > I took out the parallel processing and now I see the error: > > --- >

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC5

ny reason full dev shouldn't be working? >> >>> On Fri, Feb 24, 2017 at 9:19 PM, Casey Stella <ceste...@gmail.com> wrote: >>> >>> Sounds like a good idea to me; thanks Ryan! >>>> On Fri, Feb 24, 2017 at 21:11 Ryan Merriman <merrim...@gmail.com> wrote: &g

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC5

+1 binding Verified the signature Passed maven tests Started quick-dev, verified data in ES, kibana, and checked the topologies for errors (bro topology has parsing errors but I think a couple bad messages in bro data set is normal) Tested REPL RPMs built fine The recommended build validation

Re: [DISCUSS] Top domains enrichment config/extractor management

+1 to an Ambari view over the management UI. If we're going to go to the trouble of exposing this feature through a UI it should be intuitive and easy to use. Simply exposing a json editor in Ambari gets a -1 from me. Are we keeping track of which enrichments have been loaded? I believe the

Re: [DISCUSS] Metron Alerts UI

Related to the 'What does "Escalate" do' question, one topic that needs some discussion is how we integrate with 3rd party ticketing systems. How should we design this extension point? Some basic requirements could be that a call is made to somewhere with the alert as the payload and some kind

Re: [DISCUSS] Build Times are getting out of hand

Down to 24 minutes? Nice job. On Tue, Feb 7, 2017 at 1:49 PM, Casey Stella wrote: > I spent a minute or two looking at how we might use travis > configuration-alone to drop the wall-clock time of the build and put it up > for review at

Re: BulkMessageWriterBolt and MessageGetters

You are correct, the BulkMessageWriterBolt/MessageGetters combination is not flexible enough. You would have to modify BulkMessageWriterBolt. I have addressed this in METRON-695 which will be submitted as a PR shortly. It will be easy to do what you want after that is merged in. Ryan On Tue,

Re: [DISCUSS] Build Times are getting out of hand

Debugging integration tests in an IDE uses the same approach with our current infrastructure or with docker: start up the topology with LocalRunner. I've had mixed success with our current infrastructure. As Mike alluded to, some tests work fine (most of the parser topologies and enrichment

Re: [Discuss] Direction of metron-docker

> of small chunks of Metron, such as a single component or a single > topology? What’s doable? Other better ideas? > > Thanks, > --Matt > > > On 2/6/17, 10:03 AM, "Ryan Merriman" <merrim...@gmail.com> wrote: > > From the README: > > &quo

Re: [MENTORS] ICLA for non-committer contributions

What would our process be if someone did contribute a commit to a pull request? On Fri, Jan 27, 2017 at 4:02 PM, Casey Stella wrote: > Yes, we should definitely not destroy authorship information. To my > knowledge that hasn't happened yet and we should ensure it does not

Re: [DISCUSS] Gratuating to Apache Top Level Project

I think we're ready +1 On Wed, Jan 25, 2017 at 7:06 PM, Nick Allen wrote: > +1 I think we clearly meet all of those criteria. Glad to see the project > mature and grow. > > On Mon, Jan 23, 2017 at 7:09 PM, James Sirota wrote: > > > I think the Apache

Re: [DISCUSS] Error Indexing

. > > Jon > > On Wed, Jan 25, 2017 at 10:49 AM David Lyle <dlyle65...@gmail.com> wrote: > > RE: separate JIRA for MPack/Ansible. No objection to tracking them > separately, but for this item to be complete, you'll need both the feature > and the ability to install it. >

Re: [DISCUSS] Error Indexing

el bolt the message is passed along with no error thrown (only logged). Everywhere else I'm having trouble identifying specific fields that should be hashed. Would hashing the message in every case be acceptable? Do you know of a place where we could hash a field instead? On the topic of

Re: [DISCUSS] Error Indexing

lf) I just > haven't been able to think of another way to do this that I like better. > > Jon > > On Tue, Jan 24, 2017 at 1:13 PM Ryan Merriman <merrim...@gmail.com> wrote: > > > We also need a JIRA for any install/Ansible/MPack work needed. > > > > On Tue

Re: [DISCUSS] Error Indexing

lar function around that. > > Thanks, > James > > 24.01.2017, 10:25, "Ryan Merriman" <merrim...@gmail.com>: > > I understand what Jon is talking about. He's proposing we hash the value > > that caused the error, not necessarily the error message itself.

Re: [DISCUSS] Error Indexing

I understand what Jon is talking about. He's proposing we hash the value that caused the error, not necessarily the error message itself. For an enrichment this is easy. Just pass along the field value that failed enrichment. For other cases the field that caused the error may not be so

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

t in Ambari > > > > > > > > > > o Note this is also under the control of python code that > > we > > > > > wrote, > > > > > > > and > > > > > > > > > > this is the appropriate plac

Re: [VOTE] Reporting Issues Wiki

+1 (binding) On Mon, Jan 9, 2017 at 8:23 AM, Casey Stella wrote: > +1 (binding) > > On Fri, Jan 6, 2017 at 7:43 PM, Kyle Richardson > > wrote: > > > 0 (binding) > > > > I think it's good but it just feels a little cumbersome still. > > > > -Kyle

Re: Enrich enrichment

r <ottobackwa...@gmail.com> > > wrote: > > > > > Maybe the naming of the phases is misleading? What if you could set up > > an > > > arbitrary number of stages, with defaults? > > > > > > > > > On January 8, 2017 at 16:25:01

Re: Enrich enrichment

Hbase enrichments and geo enrichments are done in parallel so I would not expect this to work. You could do the Hbase enrichment as a threat Intel enrichment and that should work because enrichments and threat Intel are done in series. The ideal way would be to chain together Stellar

Re: [GitHub] incubator-metron issue #393: METRON-622: Create a Metron Docker Compose appl...

I would consider the topologies installed, just not running. But yes, no data flowing end to end by default. Ryan On Thu, Dec 22, 2016 at 11:42 AM, ottobackwards wrote: > Github user ottobackwards commented on the issue: > >

Re: [GitHub] incubator-metron issue #379: METRON-595: ES Writer uses more than one IP

Jonathan, That is the wrong Storm version. You need to update your Vagrant box: https://www.vagrantup.com/docs/cli/box.html. Ryan On Thu, Dec 1, 2016 at 1:50 PM, JonathanRider wrote: > Github user JonathanRider commented on the issue: > >

Re: Grok - exception accessing HDFS

My guess is that either /apps/metron/patterns/squid was somehow removed from HDFS or the HDSF url isn¹t configured properly. I believe that url comes from the fs.defaultFS property in /etc/hadoop/conf/core-site.xml. Ryan On 11/22/16, 11:14 AM, "Otto Fowler" wrote:

Re: problem with vagrant launch

Matt, Ansible is probably the most fragile of all those with respect to versions. That is likely your problem. Ryan Merriman On Mon, Nov 21, 2016 at 12:27 PM, Matt Foley <ma...@apache.org> wrote: > I had checked them, and they are all >= the specified requirement in > htt

Re: [DISCUSS] Next Release Name

+1 On Wed, Nov 9, 2016 at 4:30 PM, Casey Stella wrote: > Agreed, +1 to 0.3.0 > > On Wed, Nov 9, 2016 at 5:28 PM, zeo...@gmail.com wrote: > > > That sounds very reasonable to me. > > > > Jon > > > > On Wed, Nov 9, 2016, 17:15 James Sirota

Re: travis - how long should a build take?

The last PR I did took 43 minutes. I would restart it. On 11/9/16, 2:45 PM, "Otto Fowler" wrote: >My last pr is going on 2h.

Re: Re: Travis Logging and You

Executor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > etc etc > > I am just going to make a pass through these types and see if there is a > comm

Re: Re: Travis Logging and You

gt; wrote: > > METRON-538 > > Everyone is welcome to comment. > > > On November 3, 2016 at 12:06:28, Ryan Merriman (merrim...@gmail.com) > wrote: > > Makes sense. > > On Thu, Nov 3, 2016 at 11:03 AM, Otto Fowler <ottobackwa...@gmail.com> > wrote: >

Re: Travis Logging and You

Otto, have you started on any of this yet? Was thinking I would start with getting the log levels consistent and dig into the shutdown issues. Then we can iterate from there. Ryan On Wed, Nov 2, 2016 at 1:29 PM, Ryan Merriman <merrim...@gmail.com> wrote: > I vote for 1 logging conf

Re: [DISCUSS] Metron REST API Architecture and Design

Mon, Oct 24, 2016 at 10:51 AM, zeo...@gmail.com <zeo...@gmail.com> > > wrote: > > > > > Ok, that sounds good to me, I primarily wanted to see whether or not it > > was > > > attempted and if it hit a technical roadblock. Thanks, > > > > > > Jon > > > >

Re: Travis Logging and You

I vote for 1 logging configuration (ERROR only). Why do we want different logging in Travis vs local? If you are working on a specific component and need more verbose logging, temporarily change the log level to INFO for that component. If we get the logging in shape this will be easy to do.

Re: [DISCUSS] Build Log Usability

I definitely agree with Otto that we need to make our build logs less verbose. I think the first step is to get control of the log levels across various components. Are there specific examples of the types of messages we want to suppress? For instance:

Re: [DISCUSS] Metron REST API Architecture and Design

we should do that. On Mon, Oct 24, 2016 at 9:06 AM, Ryan Merriman <merrim...@gmail.com> wrote: > I spent some time researching the Knox documentation and building custom > services (hosted in Knox) was not well-documented. Spring is a great > choice for that and I didn't real

Re: [DISCUSS] Metron REST API Architecture and Design

pursuing Knox (per comments in > METRON-503 and then PR 316). Is there a reason for that? > > Jon > > On Fri, Oct 14, 2016 at 5:59 PM zeo...@gmail.com <zeo...@gmail.com> wrote: > > > Good question :) > > > > On Fri, Oct 14, 2016, 17:07 Ryan Merriman <merr

Re: [DISCUSS] Metron REST API Architecture and Design

on UI > - Metron Installer/upgrades > - Edge/Gateway Node for data loading > - Clients > > Also, at the end it ends mid-sentence under "Organization within Metron," > was that intended to be open ended? > > Jon > > On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman &

Re: [DISCUSS] Improving quick-dev

debug topologies > > > > > > > > > > Fortunately I think we can make this a much better experience > > without a > > > > > major effort. Here are my ideas to do this: > > > > > > > > > > - update the prereqs for VirtualBox > >

Re: [VOTE] Casey Stella as a second release manager

+1 (binding) On Fri, Oct 14, 2016 at 11:56 AM, David Lyle wrote: > +1 (binding) > > On Fri, Oct 14, 2016 at 12:47 PM, Matt Foley > wrote: > > > +1 (non-binding) > > > > On 10/14/16, 9:39 AM, "James Sirota" wrote: > > > > I

[DISCUSS] Metron REST API Architecture and Design

ideas and feedback. Ryan Merriman

[DISCUSS] Move grok pattern from HDFS to Zookeeper

this? Ryan Merriman

Re: [DISCUSS] Opinionated Data Flows

choices automatically without requiring users to have detailed knowledge of how things work internally. Ryan Merriman On Fri, Oct 7, 2016 at 7:12 AM, Nick Allen <n...@nickallen.org> wrote: > Whether it is explicit or implicit, I think that would be one of the major > benefits of having

[DISCUSS] Dockerize Metron

/tables/users created and geo data loaded Other images that could also be useful: - Images for each sensor - Ambari? - Solr Looking forward to hearing what everyone thinks. Ryan Merriman

[DISCUSS] Metron REST API Requirements

topologies, enrichments, etc) - A function that tests and runs Stellar statements against sample data. I will continue to add to these lists as I think of more. Looking forward to hearing everyone’s ideas and input. Ryan Merriman

Re: [VOTE] Adopting Bylaws

+1 (binding) On 8/17/16, 4:44 PM, "Casey Stella" wrote: >+1 (binding) > >On Wed, Aug 17, 2016 at 5:44 PM, David Lyle wrote: > >> +1 (non-binding) >> Adopt the bylaws as stated here >>

Re: [VOTE] Releasing Apache Metron 0.2.0BETA-RC2

+1 for the release On 7/7/16, 11:21 AM, "Casey Stella" wrote: >whoops, +1 for the release. > >On Thu, Jul 7, 2016 at 8:55 AM, Billie Rinaldi wrote: > >> +1 for the release. >> >> On Sat, Jul 2, 2016 at 7:58 PM, Casey Stella wrote: >>

Re: [DISCUSS] Project reorganization

think there will eventually be a need for a “Deep Analytics” project which is missing. Maybe we should include a “metron-analytics” project under “metron-platform”? If not now, in the future when we deliver more functionality in this area? Ryan Merriman On 4/19/16, 3:48 PM, "Sheetal

Re: [DISCUSS] Project reorganization

.adapters.cif.AbstractCIFAdapter.java > > > org.apache.metron.enrichment.adapters.cif.CIFHbaseAdapter.java > >org.apache.metron.enrichment.adapters.whois.WhoisHBaseAdapter.java > > >Metron-DataLoads >org.apache.metron.dataloads.cif.HBaseTableLoad.java > > &g

Re: [DISCUSS] Project reorganization

All, I put together a list of all the project java assets that details where they will be moved (or potentially deleted) as part of the project reorganization. Feedback welcome. Ryan Merriman On 4/13/16, 9:42 AM, "James Sirota" <jsir...@hortonworks.com> wrote: >I

Re: [DISCUSS] Project reorganization

irota" <jsir...@hortonworks.com> wrote: >+1 from me. > >I would also like to address the configs and make sure the configs are in >the same place. Do you have ideas on where we would put those? > >Thanks, >James > > > >On 4/13/16, 6:50 AM, "Ryan Merr

Re: [DISCUSS] Project reorganization

everything from data access (pcap service) to security and beyond. David, let’s explore the best way to leverage the dependencyManagement section in our top level pom. I think you’re on to something there. Our maven implementation needs a thorough review as well. Ryan Merriman On 4/13/16, 8

[DISCUSS] Project reorganization

ose APIs through REST or other client protocols. Could possibly depend on all other projects or separated further if version conflicts arise (separate api projects for solr and elasticsearch for example). Looking forward to hearing everyone's feedback and great ideas. Ryan Merriman

Re: [Vote] release of Metron_0.1BETA_rc7

+1 (binding) On 4/4/16, 11:16 AM, "Casey Stella" wrote: >+1 (binding) > >On Mon, Apr 4, 2016 at 12:15 PM, James Sirota >wrote: > >> + 1 (binding) >> >> >> >> >> On 4/4/16, 9:08 AM, "James Sirota" wrote: >> >> >This is a

Re: [Vote] Release of Apache Metron 0.1BETA-RC6

+1 (binding) On 3/30/16, 11:00 AM, "James Sirota" wrote: >+ 1 (binding) > > > > >On 3/30/16, 8:58 AM, "James Sirota" wrote: > >>This is a call to vote on releasing Apache Metron 0.1BETA-RC6 >> >>Full list of changes in this release: >>

Re: [VOTE] Metron_0.1BETA release

+1 > On Feb 16, 2016, at 3:18 PM, James Sirota wrote: > > I am putting up for a vote our first Apache release. Many thanks to all who > have contributed. As previously discussed we will be on a monthly release > cadence. This is the delayed Jan build (delayed due

Re: [GitHub] incubator-metron pull request: Deployment

Here is the new pull request with the modifications: https://github.com/apache/incubator-metron/pull/15 This only includes the additions our team contributed since METRON-2. On 1/26/16, 8:39 AM, "Ryan Merriman" <rmerri...@hortonworks.com> wrote: >Mark, > >We went