Re: [DISCUSS] Next Release Name

Hello James, Does that mean Metron 0.2.2 goes with HDP 2.5 by default? - Dima On 11/05/2016 06:26 AM, James Sirota wrote: > Hi Kyle, > > The HDP upgrade guide can be found here: > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-upgrade/content/ch_upgrade_2_4.html > >

Re: [DISCUSS] Next Release Name

Hi Kyle, The HDP upgrade guide can be found here: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-upgrade/content/ch_upgrade_2_4.html After executing these instructions you get to HDP 2.5 with no data loss. After that, upgrading Metron is as simple as saving the old

Re: HDFS Compression

Possibly naive question... Has there been past discussion on the use of avro for the data in HDFS? -Kyle On Tue, Oct 11, 2016 at 4:30 PM, Matt Foley wrote: > Some of the things that are desirable to do with stored data (including > those mentioned by others below): > - Use it

Re: [DISCUSS] Next Release Name

I'm a little late to the party but thought I would go ahead and throw my two cents into the mix. I share the concern around an upgrade / migration path. While I would love to see the BETA dropped sooner than later, to me, this is a game changer for people implementing Metron. I think there is a

[ANNOUNCE] Metron Apache Community Demo Recording Nov4,2016

The recording is available at: https://youtu.be/vOMZcudmlYg The meeting was a demonstration of the upcoming build. No architectural decisions about the platform were made at the meeting. The features that were demoed were: Advanced use cases of using a profiler and statistical functions to

[GitHub] incubator-metron pull request #343: METRON-548 Improve Profiler documentatio...

Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/343#discussion_r86647954 --- Diff: metron-analytics/metron-profiler/README.md --- @@ -1,16 +1,74 @@ # Metron Profiler -The Profiler is a feature

[GitHub] incubator-metron pull request #343: METRON-548 Improve Profiler documentatio...

Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/343#discussion_r86649631 --- Diff: metron-analytics/metron-profiler/README.md --- @@ -210,78 +293,52 @@ This creates a profile... * Adds the `length` field from

[GitHub] incubator-metron pull request #343: METRON-548 Improve Profiler documentatio...

Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/343#discussion_r86649179 --- Diff: metron-analytics/metron-profiler/README.md --- @@ -81,21 +139,46 @@ One or more expressions executed when a message is applied to

[GitHub] incubator-metron pull request #343: METRON-548 Improve Profiler documentatio...

Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/343#discussion_r86648252 --- Diff: metron-analytics/metron-profiler/README.md --- @@ -1,16 +1,74 @@ # Metron Profiler -The Profiler is a feature

Re: [DISCUSS] Intentional processing delays

I think you've done a good job of organizing the use cases that I have considered, and even came up with a new one seems appears valid. Jon On Fri, Nov 4, 2016, 18:01 Matt Foley wrote: > A little late to the game (look what I get for not reading my email first > thing in the

Re: [DISCUSS] Intentional processing delays

A little late to the game (look what I get for not reading my email first thing in the morning!), but in response to Jon’s initial question as to whether this would conflict with METRON-322: METRON-322, as currently being worked on, uses Tick Tuple settings to do periodic checks on internal

[GitHub] incubator-metron pull request #335: METRON-531: Ensure licenses for bundled ...

Github user joshelser commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/335#discussion_r86637885 --- Diff: metron-analytics/metron-profiler-client/src/main/resources/META-INF/LICENSE --- @@ -201,34 +201,46 @@ Apache License

[GitHub] incubator-metron pull request #335: METRON-531: Ensure licenses for bundled ...

Github user joshelser commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/335#discussion_r86638443 --- Diff: metron-analytics/metron-profiler/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,137 @@ + +metron-profiler +Copyright

[GitHub] incubator-metron pull request #335: METRON-531: Ensure licenses for bundled ...

Github user joshelser commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/335#discussion_r86638096 --- Diff: metron-analytics/metron-profiler-client/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,24 @@ + +metron-profiler-client

[GitHub] incubator-metron pull request #335: METRON-531: Ensure licenses for bundled ...

Github user joshelser commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/335#discussion_r86638908 --- Diff: metron-platform/metron-writer/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,137 @@ + +metron-writer +Copyright

[GitHub] incubator-metron pull request #335: METRON-531: Ensure licenses for bundled ...

Github user joshelser commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/335#discussion_r86638882 --- Diff: metron-platform/metron-pcap-backend/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,105 @@ + +metron-pcap-backend

[GitHub] incubator-metron pull request #335: METRON-531: Ensure licenses for bundled ...

Github user joshelser commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/335#discussion_r86638814 --- Diff: metron-platform/metron-parsers/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,99 @@ + +metron-parsers +Copyright

[GitHub] incubator-metron pull request #335: METRON-531: Ensure licenses for bundled ...

Github user joshelser commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/335#discussion_r86638699 --- Diff: metron-platform/metron-api/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,87 @@ + +metron-api +Copyright 2006-2016 The

Re: Help with custom enrichment / parser

Can you check for any exceptions in the enrichment logs using the following grep? grep --color=auto -C 3 -R -iE "exception" /var/log/storm It would also be good to know where the data is getting hung up. Can you check if you're getting tuples transferring and acking through the indexing Kafka

Hadoop Summit EU 2017

The DataWorks Summit EU 2017 (including Hadoop Summit) is going to be in Munich April 5-6 2017 . I’ve pasted the text from the CFP below. Would you like to share your knowledge with the best and brightest in the data community? If so, we encourage you to submit an abstract for DataWorks Summit

Re: [DISCUSS] Metron IRC channel

We tried using slack during the early days of the project and it was frowned upon by Apache. So we abandoned it in favor of IRC and message lists. 04.11.2016, 04:54, "zeo...@gmail.com" : > Is anybody interested in migrating this to slack? I'm personally a fan of > the benefits

Re: [DISCUSS] Intentional processing delays

You need to plan your Kafka retention and storm capacity to support spikes in traffic. You have to do this regardless of whether you delay or not. Thanks Carolyn On 11/4/16, 1:14 PM, "James Sirota" wrote: >We need to get with the Storm team and see what the new back

Re: Pittsburgh PA Meetup

Excellent. Thanks for setting that up, John. There are additional metron Meetups that will be announced soon. 04.11.2016, 03:50, "zeo...@gmail.com" : > Hi everyone, just wanted to mention that dates have been chosen for these > events. The Metron lab will be on February 9th,

Re: Re: Travis Logging and You

I think so On November 4, 2016 at 10:44:07, Ryan Merriman (merrim...@gmail.com) wrote: The maven -q flag takes care of that right? On Fri, Nov 4, 2016 at 9:39 AM, Otto Fowler wrote: > Ryan, > > Are you looking at the maven logging levels as well? The plugins etc ( >

Re: Re: Travis Logging and You

Ryan, Are you looking at the maven logging levels as well? The plugins etc ( the shading output for example )? On November 4, 2016 at 10:21:19, Otto Fowler (ottobackwa...@gmail.com) wrote: I’m guessing that if you add a duplicate appender it just replaces the one that is there of that type

Re: [DISCUSS] Intentional processing delays

I think we've come to a better way to do this which is sort of a waitUntil(exists || timeout), but the issue is checking if something exists because it requires some sort of timestamp to avoid collisions (due to source port reuse, etc.). I don't know the best way to do this offhand. Here's a

Re: [DISCUSS] Intentional processing delays

However this gets done, I think it is going to be a common problem. I think we should definitely figure this out somehow. To me this sounds like a "streaming join" problem, rather then just simply wanting to sleep. Storm has some minimal advise on doing a streaming join;

Re: [DISCUSS] Intentional processing delays

So, you want to queue up the data awaiting a key match on the enrichment data, up to a max timeout and/or buffer size? Seems like this should belong at the spout level to avoid buffer overflows, depending on how big the data sets are and how far apart the matching records/elements are spaced in

Re: [DISCUSS] Metron IRC channel

/agree On Fri, Nov 4, 2016 at 10:30 AM Casey Stella wrote: > I'm not opposed to slack; I'd just like one official avenue for realtime > interaction, so if we choose slack, I suggest shuttering irc. > > Casey > > On Fri, Nov 4, 2016 at 8:31 AM, zeo...@gmail.com

Re: [DISCUSS] Metron IRC channel

I'm not opposed to slack; I'd just like one official avenue for realtime interaction, so if we choose slack, I suggest shuttering irc. Casey On Fri, Nov 4, 2016 at 8:31 AM, zeo...@gmail.com wrote: > 1. Agreed, personally willing to accept that. > 2. I know Spot is doing it,

Re: [DISCUSS] Intentional processing delays

The reason I say spout is the naive thought that it is better to leave the ‘backup’/caching to kafka than to add it in somewhere else On November 4, 2016 at 10:28:53, Otto Fowler (ottobackwa...@gmail.com) wrote: So spout orchestration/gating? Spout checks for external state flag if CURRENT -

Re: [DISCUSS] Intentional processing delays

So spout orchestration/gating? Spout checks for external state flag if CURRENT - process if UPDATING - wait With the ingesting agent sets flag to updating when running? On November 4, 2016 at 09:29:16, zeo...@gmail.com (zeo...@gmail.com) wrote: Is there a good method (i.e. something using

Re: Re: Travis Logging and You

I’m guessing that if you add a duplicate appender it just replaces the one that is there of that type On November 4, 2016 at 10:14:08, Ryan Merriman (merrim...@gmail.com) wrote: Haha I wrote something that does the exact same thing. I added a couple extra methods to set log levels for other

Re: Re: Travis Logging and You

Haha I wrote something that does the exact same thing. I added a couple extra methods to set log levels for other logging frameworks (Log4j2 and Java logging). Good to know, I will just add on to that. On Fri, Nov 4, 2016 at 9:05 AM, Otto Fowler wrote: > Hey Ryan, > >

Re: [DISCUSS] Intentional processing delays

Very interesting use case. How big of a delay do you think you need? Can you elaborate on the two different types of data that you want to join? On Fri, Nov 4, 2016 at 9:28 AM, zeo...@gmail.com wrote: > Is there a good method (i.e. something using Stellar/ZK) to implement

Re: Re: Travis Logging and You

Hey Ryan, Take a look at the UnitTestHelper in test utils. It has methods for changing the logger verbosity. Even if it is not what we do, it is interesting. I just stubbled on it. On November 3, 2016 at 13:56:51, Otto Fowler (ottobackwa...@gmail.com) wrote: We are going to need a separate

[DISCUSS] Intentional processing delays

Is there a good method (i.e. something using Stellar/ZK) to implement an intentional processing delay to all tuples in a specific topology? I plan to do some custom enrichments, but the data used to do the enrichment *may* be ingested at roughly the same time the data to be enriched is (it also

[GitHub] incubator-metron pull request #326: METRON-510: Update elasticsearch bro tem...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/326 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

Re: [DISCUSS] Next Release Name

Jon, Thank you for your thoughts; they are appreciated and you should keep them coming. This kind of discussion is exactly why I sent out this thread. I think it's safe to say that the entire community shares your desire for Metron to be as easy to use as possible and a "data analysis platform

[GitHub] incubator-metron pull request #339: METRON-463: Pull RPMs from Remote (confi...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/339 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] incubator-metron pull request #342: METRON-536: Add ASA Parser Artifacts to ...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/342 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] incubator-metron issue #342: METRON-536: Add ASA Parser Artifacts to RPM spe...

Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/342 +1, by inspection. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

Re: [DISCUSS] Metron IRC channel

1. Agreed, personally willing to accept that. 2. I know Spot is doing it, that's the only other incubator I follow. 3. I don't do slack on my phone, but I do have IRC, so that's completely up to the person. Valid point though - breaks are good. Jon On Fri, Nov 4, 2016 at 8:27 AM Otto Fowler

Re: [DISCUSS] Metron IRC channel

I like slack. I don’t think I would complain if the group chose to move to it, but I can see some possible issues 1. The message limit. Although I currently don’t do this, I have in the past had a persistant screen session + irssi + logging to keep irc logs etc going back in time. This is

Re: [DISCUSS] Next Release Name

RE- METRON-485 I believe that there are a couple of issues here. 1. We don’t use the -w timeout parameter when killing the topologies, which means technically we may not get out cleanly. We should change this. 2. Beyond the storm timeouts monit itself has timeouts and will ‘kill’ the scripts

Re: [DISCUSS] Search Concerns

Right, that is the current state, and the short term fix here is to use ignore_above in the template, which will allow us to drop only that field and not the entire message. I'm not a huge fan of that solution, but it's better than the alternative and I've tried to make sure this issue is well

Re: [DISCUSS] Metron IRC channel

Is anybody interested in migrating this to slack? I'm personally a fan of the benefits this provides - just wanted to bring it up and see if anyone else was thinking the same thing. If not, no biggie. Jon On Thu, Sep 29, 2016 at 1:52 PM zeo...@gmail.com wrote: > +1

Re: Pittsburgh PA Meetup

Hi everyone, just wanted to mention that dates have been chosen for these events. The Metron lab will be on February 9th, and there well be a data analysis talk (specifics TBD) on January 12. Locations and other details will be added to the Meetup pages once they're finalized. Jon On Mon, Oct

Re: [DISCUSS] Next Release Name

Please understand that my points mostly relate to perception and ease of use, not what's technically possible or available. I'm coming at this as Metron should be a data analysis platform for the masses. METRON-517/542 - While I'm willing to let this one go it depends on your definition of