There is a PR up for review (
https://github.com/apache/incubator-metron/pull/518) that updates our MPack
to support a Kerberized environment. There is also a PR up for review that
adds the REST service to the MPack (
https://github.com/apache/incubator-metron/pull/500).
However, the REST
That PR was merged this morning. Do a pull and you should see it in master.
On Tue, Apr 11, 2017 at 11:44 AM, moshe jarusalem wrote:
> Hi All,
> Am I missing something basic ? or something other? Would you promptly
> redirect to a doc ?
>
> On Tue, Apr 11, 2017 at 4:08 PM,
We just finished responding to the first round of feedback so I don't think
we're that far away on METRON-623.
On Wed, Apr 5, 2017 at 3:30 PM, Matt Foley wrote:
> Totally agree would be good to have MPack support. Let’s see how it
> goes. Wouldn’t want to cut it out for the
es should
> be updated appropriately.
>
> -Taylor
>
> > On Apr 5, 2017, at 10:25 PM, James Sirota <jsir...@apache.org> wrote:
> >
> > Wanted to bump this up to the top. Did we ever get a resolution on this?
> >
> > 05.04.2017, 05:51, "Ryan Mer
be applied? If it should be applied as a
comment then why not just apply the normal header?
Thanks in advance for any help or advice you can give me. I am not able to
find a clear example or explanation and want to make sure this gets done
correctly.
Ryan Merriman
gt; Maybe, but I'd argue that we would want this to be run against a
> > > non-ansible installed cluster. For a first pass, I'd recommend just a
> > set
> > > of shell scripts utilizing the REPL and the REST API along with shell
> > > commands. Most of our capabi
I think we'll have non manual rest/web deployment soon regardless of this
discussion.
On Wed, Mar 22, 2017 at 2:00 PM, Ryan Merriman <merrim...@gmail.com> wrote:
> I don't think a cluster installed by ansible is a prerequisite to using
> ansible to integration test. They would b
nd the REST API along with shell
> commands. Most of our capabilities are quite scriptable.
>
> On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman <merrim...@gmail.com>
> wrote:
>
> > Bumping this thread. Looks like we have several +1s so I propose we move
> > to the
Bumping this thread. Looks like we have several +1s so I propose we move
to the next step. I'm anxious to get this done because these tests would
have saved me time over the last couple weeks. The management UI in
https://github.com/apache/incubator-metron/pull/484 has a set of e2e tests
being
+1 for Matt
On Tue, Mar 21, 2017 at 9:44 AM, Matt Foley wrote:
> Casey, you’ve been a great release manager. I know how much detail effort
> goes into this role.
>
> I am willing to serve as RM for the next while, if the community would
> like. I was the RM for Hadoop for
The delete topic function is Kafka has had some issues in the past. I
don't think it's critical we expose that through the REST API so I propose
we just take it out. Any objections?
On Thu, Mar 16, 2017 at 3:52 PM, cestella wrote:
> Github user cestella commented on the
+1 (binding)
> On Mar 13, 2017, at 6:05 PM, Casey Stella wrote:
>
> +1 (binding)
>
>> On Mon, Mar 13, 2017 at 6:37 PM, James Sirota wrote:
>>
>> +1 (binding)
>>
>> 13.03.2017, 15:37, "James Sirota" :
>>> Do we feel it's time for us
+1 (binding)
> On Mar 13, 2017, at 6:04 PM, Justin Leet wrote:
>
> +1 (non-binding)
>
>> On Mon, Mar 13, 2017 at 6:35 PM, zeo...@gmail.com wrote:
>>
>> +1 (non-binding)
>>
>>> On Mon, Mar 13, 2017 at 6:34 PM James Sirota wrote:
I will get that added to the README.
On Wed, Mar 8, 2017 at 1:22 PM, Otto Fowler wrote:
> Never mind. found it in the tests.
>
>
>
> On March 8, 2017 at 14:18:22, Otto Fowler (ottobackwa...@gmail.com) wrote:
>
> I’m trying to run the metron-interface/metron-rest
I have a good understanding of this since I spent a good amount of time
troubleshooting it. Let me attempt to explain it. Hopefully if everyone
clearly understands the core issue we can put our heads together for a
solution.
The metron-common module shades guava and is installed as an uber jar
Here is an explanation:
https://github.com/apache/incubator-metron/pull/316#issuecomment-282791185
What exactly do you mean by "build is failing". Are you trying to spin it
up in Intellij? If so you'll need to add the metron-common jar that is
installed in your local Maven repo (the uber jar
+1, great idea. At some point our manual testing checklist is going to
grow large enough that we'll need to move to something more automated.
We're probably already there.
I very much agree with Justin's concern. Building and running
unit/integration tests takes a long time right now and this
I run into this error every now and then. If I free up space on my hard drive
it goes away.
> On Feb 27, 2017, at 8:31 PM, Otto Fowler wrote:
>
> I took out the parallel processing and now I see the error:
>
> ---
>
ny reason full dev shouldn't be working?
>>
>>> On Fri, Feb 24, 2017 at 9:19 PM, Casey Stella <ceste...@gmail.com> wrote:
>>>
>>> Sounds like a good idea to me; thanks Ryan!
>>>> On Fri, Feb 24, 2017 at 21:11 Ryan Merriman <merrim...@gmail.com> wrote:
&g
+1 binding
Verified the signature
Passed maven tests
Started quick-dev, verified data in ES, kibana, and checked the topologies
for errors (bro topology has parsing errors but I think a couple bad
messages in bro data set is normal)
Tested REPL
RPMs built fine
The recommended build validation
+1 to an Ambari view over the management UI. If we're going to go to the
trouble of exposing this feature through a UI it should be intuitive and
easy to use. Simply exposing a json editor in Ambari gets a -1 from me.
Are we keeping track of which enrichments have been loaded? I believe the
Related to the 'What does "Escalate" do' question, one topic that needs
some discussion is how we integrate with 3rd party ticketing systems. How
should we design this extension point? Some basic requirements could be
that a call is made to somewhere with the alert as the payload and some
kind
Down to 24 minutes? Nice job.
On Tue, Feb 7, 2017 at 1:49 PM, Casey Stella wrote:
> I spent a minute or two looking at how we might use travis
> configuration-alone to drop the wall-clock time of the build and put it up
> for review at
You are correct, the BulkMessageWriterBolt/MessageGetters combination is
not flexible enough. You would have to modify BulkMessageWriterBolt. I
have addressed this in METRON-695 which will be submitted as a PR shortly.
It will be easy to do what you want after that is merged in.
Ryan
On Tue,
Debugging integration tests in an IDE uses the same approach with our
current infrastructure or with docker: start up the topology with
LocalRunner. I've had mixed success with our current infrastructure. As
Mike alluded to, some tests work fine (most of the parser topologies and
enrichment
> of small chunks of Metron, such as a single component or a single
> topology? What’s doable? Other better ideas?
>
> Thanks,
> --Matt
>
>
> On 2/6/17, 10:03 AM, "Ryan Merriman" <merrim...@gmail.com> wrote:
>
> From the README:
>
> &quo
What would our process be if someone did contribute a commit to a pull
request?
On Fri, Jan 27, 2017 at 4:02 PM, Casey Stella wrote:
> Yes, we should definitely not destroy authorship information. To my
> knowledge that hasn't happened yet and we should ensure it does not
I think we're ready +1
On Wed, Jan 25, 2017 at 7:06 PM, Nick Allen wrote:
> +1 I think we clearly meet all of those criteria. Glad to see the project
> mature and grow.
>
> On Mon, Jan 23, 2017 at 7:09 PM, James Sirota wrote:
>
> > I think the Apache
.
>
> Jon
>
> On Wed, Jan 25, 2017 at 10:49 AM David Lyle <dlyle65...@gmail.com> wrote:
>
> RE: separate JIRA for MPack/Ansible. No objection to tracking them
> separately, but for this item to be complete, you'll need both the feature
> and the ability to install it.
>
el bolt the
message is passed along with no error thrown (only logged). Everywhere
else I'm having trouble identifying specific fields that should be hashed.
Would hashing the message in every case be acceptable? Do you know of a
place where we could hash a field instead? On the topic of
lf) I just
> haven't been able to think of another way to do this that I like better.
>
> Jon
>
> On Tue, Jan 24, 2017 at 1:13 PM Ryan Merriman <merrim...@gmail.com> wrote:
>
> > We also need a JIRA for any install/Ansible/MPack work needed.
> >
> > On Tue
lar function around that.
>
> Thanks,
> James
>
> 24.01.2017, 10:25, "Ryan Merriman" <merrim...@gmail.com>:
> > I understand what Jon is talking about. He's proposing we hash the value
> > that caused the error, not necessarily the error message itself.
I understand what Jon is talking about. He's proposing we hash the value
that caused the error, not necessarily the error message itself. For an
enrichment this is easy. Just pass along the field value that failed
enrichment. For other cases the field that caused the error may not be so
t in Ambari
> > > > > > > > > > o Note this is also under the control of python code that
> > we
> > > > > wrote,
> > > > > > > and
> > > > > > > > > > this is the appropriate plac
+1 (binding)
On Mon, Jan 9, 2017 at 8:23 AM, Casey Stella wrote:
> +1 (binding)
>
> On Fri, Jan 6, 2017 at 7:43 PM, Kyle Richardson >
> wrote:
>
> > 0 (binding)
> >
> > I think it's good but it just feels a little cumbersome still.
> >
> > -Kyle
r <ottobackwa...@gmail.com>
> > wrote:
> >
> > > Maybe the naming of the phases is misleading? What if you could set up
> > an
> > > arbitrary number of stages, with defaults?
> > >
> > >
> > > On January 8, 2017 at 16:25:01
Hbase enrichments and geo enrichments are done in parallel so I would not
expect this to work. You could do the Hbase enrichment as a threat Intel
enrichment and that should work because enrichments and threat Intel are done
in series.
The ideal way would be to chain together Stellar
I would consider the topologies installed, just not running. But yes, no
data flowing end to end by default.
Ryan
On Thu, Dec 22, 2016 at 11:42 AM, ottobackwards wrote:
> Github user ottobackwards commented on the issue:
>
>
Jonathan,
That is the wrong Storm version. You need to update your Vagrant box:
https://www.vagrantup.com/docs/cli/box.html.
Ryan
On Thu, Dec 1, 2016 at 1:50 PM, JonathanRider wrote:
> Github user JonathanRider commented on the issue:
>
>
My guess is that either /apps/metron/patterns/squid was somehow removed
from HDFS or the HDSF url isn¹t configured properly. I believe that url
comes from the fs.defaultFS property in /etc/hadoop/conf/core-site.xml.
Ryan
On 11/22/16, 11:14 AM, "Otto Fowler" wrote:
Matt,
Ansible is probably the most fragile of all those with respect to
versions. That is likely your problem.
Ryan Merriman
On Mon, Nov 21, 2016 at 12:27 PM, Matt Foley <ma...@apache.org> wrote:
> I had checked them, and they are all >= the specified requirement in
> htt
+1
On Wed, Nov 9, 2016 at 4:30 PM, Casey Stella wrote:
> Agreed, +1 to 0.3.0
>
> On Wed, Nov 9, 2016 at 5:28 PM, zeo...@gmail.com wrote:
>
> > That sounds very reasonable to me.
> >
> > Jon
> >
> > On Wed, Nov 9, 2016, 17:15 James Sirota
The last PR I did took 43 minutes. I would restart it.
On 11/9/16, 2:45 PM, "Otto Fowler" wrote:
>My last pr is going on 2h.
Executor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> etc etc
>
> I am just going to make a pass through these types and see if there is a
> comm
gt; wrote:
>
> METRON-538
>
> Everyone is welcome to comment.
>
>
> On November 3, 2016 at 12:06:28, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> Makes sense.
>
> On Thu, Nov 3, 2016 at 11:03 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
Otto, have you started on any of this yet? Was thinking I would start with
getting the log levels consistent and dig into the shutdown issues. Then
we can iterate from there.
Ryan
On Wed, Nov 2, 2016 at 1:29 PM, Ryan Merriman <merrim...@gmail.com> wrote:
> I vote for 1 logging conf
Mon, Oct 24, 2016 at 10:51 AM, zeo...@gmail.com <zeo...@gmail.com>
> > wrote:
> >
> > > Ok, that sounds good to me, I primarily wanted to see whether or not it
> > was
> > > attempted and if it hit a technical roadblock. Thanks,
> > >
> > > Jon
> > >
>
I vote for 1 logging configuration (ERROR only). Why do we want different
logging in Travis vs local? If you are working on a specific component and
need more verbose logging, temporarily change the log level to INFO for
that component. If we get the logging in shape this will be easy to do.
I definitely agree with Otto that we need to make our build logs less
verbose. I think the first step is to get control of the log levels across
various components. Are there specific examples of the types of messages
we want to suppress? For instance:
we should do that.
On Mon, Oct 24, 2016 at 9:06 AM, Ryan Merriman <merrim...@gmail.com> wrote:
> I spent some time researching the Knox documentation and building custom
> services (hosted in Knox) was not well-documented. Spring is a great
> choice for that and I didn't real
pursuing Knox (per comments in
> METRON-503 and then PR 316). Is there a reason for that?
>
> Jon
>
> On Fri, Oct 14, 2016 at 5:59 PM zeo...@gmail.com <zeo...@gmail.com> wrote:
>
> > Good question :)
> >
> > On Fri, Oct 14, 2016, 17:07 Ryan Merriman <merr
on UI
> - Metron Installer/upgrades
> - Edge/Gateway Node for data loading
> - Clients
>
> Also, at the end it ends mid-sentence under "Organization within Metron,"
> was that intended to be open ended?
>
> Jon
>
> On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman &
debug topologies
> > > > >
> > > > > Fortunately I think we can make this a much better experience
> > without a
> > > > > major effort. Here are my ideas to do this:
> > > > >
> > > > > - update the prereqs for VirtualBox
> >
+1 (binding)
On Fri, Oct 14, 2016 at 11:56 AM, David Lyle wrote:
> +1 (binding)
>
> On Fri, Oct 14, 2016 at 12:47 PM, Matt Foley
> wrote:
>
> > +1 (non-binding)
> >
> > On 10/14/16, 9:39 AM, "James Sirota" wrote:
> >
> > I
ideas and feedback.
Ryan Merriman
this?
Ryan Merriman
choices
automatically without requiring users to have detailed knowledge of how
things work internally.
Ryan Merriman
On Fri, Oct 7, 2016 at 7:12 AM, Nick Allen <n...@nickallen.org> wrote:
> Whether it is explicit or implicit, I think that would be one of the major
> benefits of having
/tables/users created and geo data loaded
Other images that could also be useful:
- Images for each sensor
- Ambari?
- Solr
Looking forward to hearing what everyone thinks.
Ryan Merriman
topologies, enrichments, etc)
- A function that tests and runs Stellar statements against sample data.
I will continue to add to these lists as I think of more. Looking forward
to hearing everyone’s ideas and input.
Ryan Merriman
+1 (binding)
On 8/17/16, 4:44 PM, "Casey Stella" wrote:
>+1 (binding)
>
>On Wed, Aug 17, 2016 at 5:44 PM, David Lyle wrote:
>
>> +1 (non-binding)
>> Adopt the bylaws as stated here
>>
+1 for the release
On 7/7/16, 11:21 AM, "Casey Stella" wrote:
>whoops, +1 for the release.
>
>On Thu, Jul 7, 2016 at 8:55 AM, Billie Rinaldi wrote:
>
>> +1 for the release.
>>
>> On Sat, Jul 2, 2016 at 7:58 PM, Casey Stella wrote:
>>
think there will eventually be a need for a “Deep
Analytics” project which is missing. Maybe we should include a
“metron-analytics” project under “metron-platform”? If not now, in the
future when we deliver more functionality in this area?
Ryan Merriman
On 4/19/16, 3:48 PM, "Sheetal
.adapters.cif.AbstractCIFAdapter.java
>
>
> org.apache.metron.enrichment.adapters.cif.CIFHbaseAdapter.java
>
>org.apache.metron.enrichment.adapters.whois.WhoisHBaseAdapter.java
>
>
>Metron-DataLoads
>org.apache.metron.dataloads.cif.HBaseTableLoad.java
>
>
&g
All,
I put together a list of all the project java assets that details where
they will be moved (or potentially deleted) as part of the project
reorganization. Feedback welcome.
Ryan Merriman
On 4/13/16, 9:42 AM, "James Sirota" <jsir...@hortonworks.com> wrote:
>I
irota" <jsir...@hortonworks.com> wrote:
>+1 from me.
>
>I would also like to address the configs and make sure the configs are in
>the same place. Do you have ideas on where we would put those?
>
>Thanks,
>James
>
>
>
>On 4/13/16, 6:50 AM, "Ryan Merr
everything from
data access (pcap service) to security and beyond.
David, let’s explore the best way to leverage the dependencyManagement
section in our top level pom. I think you’re on to something there. Our
maven implementation needs a thorough review as well.
Ryan Merriman
On 4/13/16, 8
ose APIs through REST or other client protocols. Could possibly depend on
all other projects or separated further if version conflicts arise (separate
api projects for solr and elasticsearch for example).
Looking forward to hearing everyone's feedback and great ideas.
Ryan Merriman
+1 (binding)
On 4/4/16, 11:16 AM, "Casey Stella" wrote:
>+1 (binding)
>
>On Mon, Apr 4, 2016 at 12:15 PM, James Sirota
>wrote:
>
>> + 1 (binding)
>>
>>
>>
>>
>> On 4/4/16, 9:08 AM, "James Sirota" wrote:
>>
>> >This is a
+1 (binding)
On 3/30/16, 11:00 AM, "James Sirota" wrote:
>+ 1 (binding)
>
>
>
>
>On 3/30/16, 8:58 AM, "James Sirota" wrote:
>
>>This is a call to vote on releasing Apache Metron 0.1BETA-RC6
>>
>>Full list of changes in this release:
>>
+1
> On Feb 16, 2016, at 3:18 PM, James Sirota wrote:
>
> I am putting up for a vote our first Apache release. Many thanks to all who
> have contributed. As previously discussed we will be on a monthly release
> cadence. This is the delayed Jan build (delayed due
Here is the new pull request with the modifications:
https://github.com/apache/incubator-metron/pull/15
This only includes the additions our team contributed since METRON-2.
On 1/26/16, 8:39 AM, "Ryan Merriman" <rmerri...@hortonworks.com> wrote:
>Mark,
>
>We went
71 matches
Mail list logo