Simply as a unique identifier of the original information which is failing some step, and thus giving you something to key in on and create a count of unique events and prioritize issues without the concern of cyclical issues (if the issue is with indexing a specific message, and you try to index it again, it will just fail in a loop).
Jon On Wed, Feb 1, 2017 at 6:59 AM Dima Kovalyov <dima.koval...@sstech.us> wrote: > That's a great topic of discussion. > > Throughout the thread the idea of having hash of the message that failed > is changed, can someone please explain why do you plan to use this hash > and how? > > - Dima > > On 02/01/2017 06:23 AM, zeo...@gmail.com wrote: > > After thinking on this for a few days I recant my previous suggestion of > > TupleHash256. It's still a bit early for SHA-3 - no good reference > > implementations/libraries exist (I did some searching and emailing), it > is > > optimized for hardware but no hardware implementation is widely > accessible, > > FIPS 140-3 is still not close to finalized, etc. > > > > I think we could simulate the benefits of tuplehash by sorting the > tuples, > > then doing SHA-256(len(tuple1) | tuple1 | ... | len(tuplen) | tuplen). > > Happy to entertain opposing thoughts, such as BLAKE2, etc. but with the > > likely users of Metron, I think sticking with FIPS 140-2 is a solid > choice. > > > > Jon > > > > On Thu, Jan 26, 2017, 11:23 AM zeo...@gmail.com <zeo...@gmail.com> > wrote: > > > > So one more thing regarding why I think we should throw an exception on a > > failed enrichment. If we do make something like username a constant > field, > > in cases where that is used to calculate rawMessage_hash, if it fails to > > enrich, the hash would be different compared to when it succeeds. Of > > course I think the initial intent of adding username as a constant field > > would be to handle it in the parsers, where that information is provided > in > > the messages themselves, but how would Threat Intel know the difference? > > In my environment I am looking forward to a streaming enrichment that > adds > > the username, where applicable, anywhere I have an IP. > > > > My hesitant suggestion for a hashing algorithm would be to use > > TupleHash256, as it is a NIST-provided implementation of SHA-3 (using > > cSHAKE) for this use case. Details here > > < > http://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-185.pdf>. > > However, I haven't been able to find a reference implementation of this > in > > any language, so that's a bit of a downside. A more general SHA3-256 > > implementation where we handle ordering could work as well, but would be > > significantly less optimal. > > > > Jon > > > > On Thu, Jan 26, 2017 at 10:20 AM Ryan Merriman <merrim...@gmail.com> > wrote: > > > > Jon, I misread the code in the GenericEnrichmentBolt. The error is > > forwarded on so no issues there. > > > > Defaulting to the common fields makes sense. I will dig into the > > GenericEnrichmentBolt more, maybe there is a way to get the error fields > > without having to significantly change things. Any opinion on a hashing > > algorithm? > > > > On Wed, Jan 25, 2017 at 9:37 PM, zeo...@gmail.com <zeo...@gmail.com> > wrote: > > > >> Although hashing the whole message is better than nothing, it misses a > lot > >> of the benefits we could get. > >> > >> While I'd love to have consistency for this field across all of the > >> different error.types, it appears that may not be reasonably possible > >> because of the parsers. So, how about something like hash all of the > >> constant > >> fields > >> <https://github.com/apache/incubator-metron/blob/master/ > >> metron-platform/metron-common/src/main/java/org/apache/ > >> metron/common/Constants.java> > >> excluding > >> timestamp and original_string unless it is a parser, in which case hash > > the > >> entire message? This gives us some measure of event uniqueness and it > can > >> grow as we define additional constant fields (I recall discussing with > >> someone else on the list regarding expanding those standard fields to > >> include things like usernames but I can't find the specific email > >> exchange). > >> > >> Because some enrichments can be heavily relied on, I think it makes > sense > >> to put a message onto the error queue when it throws an exception. Not > >> only does this help troubleshoot edge cases, but it makes issues more > >> obvious when assembling a new enrichment in dev/test. I can't think of > a > >> scenario currently where an enrichment would only be "best effort" and > > that > >> I wouldn't want that error indexed and retrievable. However, this gets > >> interesting when talking about the various options to solve the "Enrich > >> enrichment" discussion from earlier in the month. We can keep that part > > of > >> this separate though, as I don't think that's being actively pursued > right > >> now. > >> > >> Jon > >> > >> On Wed, Jan 25, 2017 at 10:49 AM David Lyle <dlyle65...@gmail.com> > wrote: > >> > >> RE: separate JIRA for MPack/Ansible. No objection to tracking them > >> separately, but for this item to be complete, you'll need both the > feature > >> and the ability to install it. > >> > >> -D... > >> > >> > >> On Tue, Jan 24, 2017 at 5:33 PM, Ryan Merriman <merrim...@gmail.com> > >> wrote: > >> > >>> Assuming we're going to write all errors to a single error topic, I > > think > >>> it makes sense to agree on an error message schema and handle errors > >> across > >>> the 3 different topologies in the same way with a single > implementation. > >>> The implementation in ParserBolt (ErrorUtils.handleError) produces the > >> most > >>> verbose error object so I think it's a good candidate for the single > >>> implementation. Here is the message structure it currently produces: > >>> > >>> { > >>> "exception": "java.lang.Exception: there was an error", > >>> "hostname": "host", > >>> "stack": "java.lang.Exception: ...", > >>> "time": 1485295416563, > >>> "message": "there was an error", > >>> "rawMessage": "raw message", > >>> "rawMessage_bytes": [], > >>> "source.type": "bro_error" > >>> } > >>> > >>> From our discussion so far we need to add a couple fields: an error > > type > >>> and hash id. Adding these to the message looks like: > >>> > >>> { > >>> "exception": "java.lang.Exception: there was an error", > >>> "hostname": "host", > >>> "stack": "java.lang.Exception: ...", > >>> "time": 1485295416563, > >>> "message": "there was an error", > >>> "rawMessage": "raw message", > >>> "rawMessage_bytes": [], > >>> "source.type": "bro_error", > >>> "error.type": "parser_error", > >>> "rawMessage_hash": "dde41b9920954f94066daf6291fb58a9" > >>> } > >>> > >>> We should also consider expanding the error types I listed earlier. > >>> Instead of just having "indexing_error" we could have > >>> "elasticsearch_indexing_error", "hdfs_indexing_error" and so on. > >>> > >>> Jon, if an exception happens in an enrichment or threat intel bolt the > >>> message is passed along with no error thrown (only logged). Everywhere > >>> else I'm having trouble identifying specific fields that should be > >> hashed. > >>> Would hashing the message in every case be acceptable? Do you know of > a > >>> place where we could hash a field instead? On the topic of exceptions > > in > >>> enrichments, are we ok with an error only being logged and not added to > >> the > >>> message or emitted to the error queue? > >>> > >>> > >>> > >>> On Tue, Jan 24, 2017 at 3:10 PM, Ryan Merriman <merrim...@gmail.com> > >>> wrote: > >>> > >>>> That use case makes sense to me. I don't think it will require that > >> much > >>>> additional effort either. > >>>> > >>>> On Tue, Jan 24, 2017 at 1:02 PM, zeo...@gmail.com <zeo...@gmail.com> > >>>> wrote: > >>>> > >>>>> Regarding error vs validation - Either way I'm not very concerned. I > >>>>> initially assumed they would be combined and agree with that > > approach, > >>> but > >>>>> splitting them out isn't a very big deal to me either. > >>>>> > >>>>> Re: Ryan. Yes, exactly. In the case of a parser issue (or anywhere > >>> else > >>>>> where it's not possible to pick out the exact thing causing the > > issue) > >>> it > >>>>> would be a hash of the complete message. > >>>>> > >>>>> Regarding the architecture, I mostly agree with James except that I > >>> think > >>>>> step 3 needs to also be able to somehow group errors via the original > >>>>> data (identify > >>>>> replays, identify repeat issues with data in a specific field, issues > >>> with > >>>>> consistently different data, etc.). This is essentially the first > >> step > >>> of > >>>>> troubleshooting, which I assume you are doing if you're looking at > > the > >>>>> error dashboard. > >>>>> > >>>>> If the hash gets moved out of the initial implementation, I'm fairly > >>>>> certain you lose this ability. The point here isn't to handle long > >>> fields > >>>>> (although that's a benefit of this approach), it's to attach a unique > >>>>> identifier to the error/validation issue message that links it to the > >>>>> original problem. I'd be happy to consider alternative solutions to > >>> this > >>>>> problem (for instance, actually sending across the data itself) I > > just > >>>>> haven't been able to think of another way to do this that I like > >> better. > >>>>> Jon > >>>>> > >>>>> On Tue, Jan 24, 2017 at 1:13 PM Ryan Merriman <merrim...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> We also need a JIRA for any install/Ansible/MPack work needed. > >>>>>> > >>>>>> On Tue, Jan 24, 2017 at 12:06 PM, James Sirota <jsir...@apache.org> > >>>>> wrote: > >>>>>>> Now that I had some time to think about it I would collapse all > >>> error > >>>>> and > >>>>>>> validation topics into one. We can differentiate between > >> different > >>>>> views > >>>>>>> of the data (split by error source etc) via Kibana dashboards. I > >>>>> would > >>>>>>> implement this feature incrementally. First I would modify all > >> the > >>>>> bolts > >>>>>>> to log to a single topic. Second, I would get the error indexing > >>>>> done by > >>>>>>> attaching the indexing topology to the error topic. Third I would > >>>>> create > >>>>>>> the necessary dashboards to view errors and validation failures > > by > >>>>>> source. > >>>>>>> Lastly, I would file a follow-on JIRA to introduce hashing of > >> errors > >>>>> or > >>>>>>> fields that are too long. It seems like a separate feature that > >> we > >>>>> need > >>>>>> to > >>>>>>> think through. We may need a stellar function around that. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> James > >>>>>>> > >>>>>>> 24.01.2017, 10:25, "Ryan Merriman" <merrim...@gmail.com>: > >>>>>>>> I understand what Jon is talking about. He's proposing we hash > >> the > >>>>>> value > >>>>>>>> that caused the error, not necessarily the error message > > itself. > >>>>> For an > >>>>>>>> enrichment this is easy. Just pass along the field value that > >>> failed > >>>>>>>> enrichment. For other cases the field that caused the error may > >>> not > >>>>> be > >>>>>> so > >>>>>>>> obvious. Take parser validation for example. The message is > >>>>> validated > >>>>>> as > >>>>>>>> a whole and it may not be easy to determine which field is the > >>>>> cause. > >>>>>> In > >>>>>>>> that case would a hash of the whole message work? > >>>>>>>> > >>>>>>>> There is a broader architectural discussion that needs to > > happen > >>>>> before > >>>>>>> we > >>>>>>>> can implement this. Currently we have an indexing topology that > >>>>> reads > >>>>>>> from > >>>>>>>> 1 topic and writes messages to ES but errors are written to > >>> several > >>>>>>>> different topics: > >>>>>>>> > >>>>>>>> - parser_error > >>>>>>>> - parser_invalid > >>>>>>>> - enrichments_error > >>>>>>>> - threatintel_error > >>>>>>>> - indexing_error > >>>>>>>> > >>>>>>>> I can see 4 possible approaches to implementing this: > >>>>>>>> > >>>>>>>> 1. Create an index topology for each error topic > >>>>>>>> 1. Good because we can easily reuse the indexing topology > >>> and > >>>>>> would > >>>>>>>> require the least development effort > >>>>>>>> 2. Bad because it would consume a lot of extra worker > >> slots > >>>>>>>> 2. Move the topic name into the error JSON message as a new > >>>>>>> "error_type" > >>>>>>>> field and write all messages to the indexing topic > >>>>>>>> 1. Good because we don't need to create a new topology > >>>>>>>> 2. Bad because we would be flowing data and errors > > through > >>> the > >>>>>> same > >>>>>>>> topology. A spike in errors could affect message > > indexing. > >>>>>>>> 3. Compromise between 1 and 2. Create another indexing > >> topology > >>>>> that > >>>>>>> is > >>>>>>>> dedicated to indexing errors. Move the topic name into the > >>> error > >>>>>> JSON > >>>>>>>> message as a new "error_type" field and write all errors to > > a > >>>>> single > >>>>>>> error > >>>>>>>> topic. > >>>>>>>> 4. Write a completely new topology with multiple spouts (1 > >> for > >>>>> each > >>>>>>>> error type listed above) that all feed into a single > >>>>>>> BulkMessageWriterBolt. > >>>>>>>> 1. Good because the current topologies would not need to > >>>>> change > >>>>>>>> 2. Bad because it would require the most development > >> effort, > >>>>>> would > >>>>>>>> not reuse existing topologies and takes up more worker > >> slots > >>>>>> than 3 > >>>>>>>> Are there other approaches I haven't thought of? I think 1 and > > 2 > >>> are > >>>>>> off > >>>>>>>> the table because they are shortcuts and not good long-term > >>>>> solutions. > >>>>>> 3 > >>>>>>>> would be my choice because it introduces less complexity than > > 4. > >>>>>>> Thoughts? > >>>>>>>> Ryan > >>>>>>>> > >>>>>>>> On Mon, Jan 23, 2017 at 5:44 PM, zeo...@gmail.com < > >>> zeo...@gmail.com > >>>>>>> wrote: > >>>>>>>>> In that case the hash would be of the value in the IP field, > >>> such > >>>>> as > >>>>>>>>> sha3(8.8.8.8). > >>>>>>>>> > >>>>>>>>> Jon > >>>>>>>>> > >>>>>>>>> On Mon, Jan 23, 2017, 6:41 PM James Sirota < > >> jsir...@apache.org> > >>>>>> wrote: > >>>>>>>>> > Jon, > >>>>>>>>> > > >>>>>>>>> > I am still not entirely following why we would want to use > >>>>> hashing. > >>>>>>> For > >>>>>>>>> > example if my error is "Your IP field is invalid and failed > >>>>>>> validation" > >>>>>>>>> > hashing this error string will always result in the same > >> hash. > >>>>> Why > >>>>>>> not > >>>>>>>>> > just use the actual error string? Can you provide an > > example > >>>>> where > >>>>>>> you > >>>>>>>>> > would use it? > >>>>>>>>> > > >>>>>>>>> > Thanks, > >>>>>>>>> > James > >>>>>>>>> > > >>>>>>>>> > 23.01.2017, 16:29, "zeo...@gmail.com" <zeo...@gmail.com>: > >>>>>>>>> > > For 1 - I'm good with that. > >>>>>>>>> > > > >>>>>>>>> > > I'm talking about hashing the relevant content itself not > >>> the > >>>>>>> error. > >>>>>>>>> Some > >>>>>>>>> > > benefits are (1) minimize load on search index (there's > >>>>> minimal > >>>>>>> benefit > >>>>>>>>> > in > >>>>>>>>> > > spending the CPU and disk to keep it at full fidelity > >>>>> (tokenize > >>>>>> and > >>>>>>>>> > store)) > >>>>>>>>> > > (2) provide something to key on for dashboards (assuming > > a > >>>>> good > >>>>>>> hash > >>>>>>>>> > > algorithm that avoids collisions and is second preimage > >>>>>> resistant) > >>>>>>> and > >>>>>>>>> > (3) > >>>>>>>>> > > specific to errors, if the issue is that it failed to > >>> index, a > >>>>>> hash > >>>>>>>>> gives > >>>>>>>>> > > us some protection that the issue will not occur twice. > >>>>>>>>> > > > >>>>>>>>> > > Jon > >>>>>>>>> > > > >>>>>>>>> > > On Mon, Jan 23, 2017, 2:47 PM James Sirota < > >>>>> jsir...@apache.org> > >>>>>>> wrote: > >>>>>>>>> > > > >>>>>>>>> > > Jon, > >>>>>>>>> > > > >>>>>>>>> > > With regards to 1, collapsing to a single dashboard for > >> each > >>>>>> would > >>>>>>> be > >>>>>>>>> > > fine. So we would have one error index and one "failed to > >>>>>> validate" > >>>>>>>>> > > index. The distinction is that errors would be things > > that > >>>>> went > >>>>>>> wrong > >>>>>>>>> > > during stream processing (failed to parse, etc...), while > >>>>>>> validation > >>>>>>>>> > > failures are messages that explicitly failed stellar > >>>>>>> validation/schema > >>>>>>>>> > > enforcement. There should be relatively few of the second > >>>>> type. > >>>>>>>>> > > > >>>>>>>>> > > With respect to 3, why do you want the error hashed? Why > >> not > >>>>> just > >>>>>>>>> search > >>>>>>>>> > > for the error text? > >>>>>>>>> > > > >>>>>>>>> > > Thanks, > >>>>>>>>> > > James > >>>>>>>>> > > > >>>>>>>>> > > 20.01.2017, 14:01, "zeo...@gmail.com" <zeo...@gmail.com>: > >>>>>>>>> > >> As someone who currently fills the platform engineer > >> role, > >>> I > >>>>> can > >>>>>>> give > >>>>>>>>> > this > >>>>>>>>> > >> idea a huge +1. My thoughts: > >>>>>>>>> > >> > >>>>>>>>> > >> 1. I think it depends on exactly what data is pushed > > into > >>> the > >>>>>>> index > >>>>>>>>> > (#3). > >>>>>>>>> > >> However, assuming the errors you proposed recording, I > >>> can't > >>>>> see > >>>>>>> huge > >>>>>>>>> > >> benefits to having more than one dashboard. I would be > >>> happy > >>>>> to > >>>>>> be > >>>>>>>>> > >> persuaded otherwise. > >>>>>>>>> > >> > >>>>>>>>> > >> 2. I would say yes, storing the errors in HDFS in > >> addition > >>> to > >>>>>>>>> indexing > >>>>>>>>> > is > >>>>>>>>> > >> a good thing. Using METRON-510 > >>>>>>>>> > >> <https://issues.apache.org/jira/browse/METRON-510> as a > >>> case > >>>>>>> study, > >>>>>>>>> > there > >>>>>>>>> > >> is the potential in this environment for > >>> attacker-controlled > >>>>>> data > >>>>>>> to > >>>>>>>>> > > > >>>>>>>>> > > result > >>>>>>>>> > >> in processing errors which could be a method of evading > >>>>> security > >>>>>>>>> > >> monitoring. Once an attack is identified, the long term > >>> HDFS > >>>>>>> storage > >>>>>>>>> > would > >>>>>>>>> > >> allow better historical analysis for > >>> low-and-slow/persistent > >>>>>>> attacks > >>>>>>>>> > (I'm > >>>>>>>>> > >> thinking of a method of data exfil that also won't > >>>>> successfully > >>>>>>> get > >>>>>>>>> > stored > >>>>>>>>> > >> in Lucene, but is hard to identify over a short period > > of > >>>>> time). > >>>>>>>>> > >> - Along this line, I think that there are various parts > >> of > >>>>>> Metron > >>>>>>>>> > (this > >>>>>>>>> > >> included) which could benefit from having method of > >>>>> configuring > >>>>>>> data > >>>>>>>>> > aging > >>>>>>>>> > >> by bucket in HDFS (Following Nick's comments here > >>>>>>>>> > >> <https://issues.apache.org/jira/browse/METRON-477>). > >>>>>>>>> > >> > >>>>>>>>> > >> 3. I would potentially add a hash of the content that > >>> failed > >>>>>>>>> > validation to > >>>>>>>>> > >> help identify repeats over time with less of a concern > >> that > >>>>>> you'd > >>>>>>>>> have > >>>>>>>>> > > > >>>>>>>>> > > back > >>>>>>>>> > >> to back failures (i.e. instead of storing the value > >>> itself). > >>>>>>>>> > Additionally, > >>>>>>>>> > >> I think it's helpful to be able to search all times > > there > >>>>> was an > >>>>>>>>> > indexing > >>>>>>>>> > >> error (instead of it hitting the catch-all). > >>>>>>>>> > >> > >>>>>>>>> > >> Jon > >>>>>>>>> > >> > >>>>>>>>> > >> On Fri, Jan 20, 2017 at 1:17 PM James Sirota < > >>>>>> jsir...@apache.org> > >>>>>>>>> > wrote: > >>>>>>>>> > >> > >>>>>>>>> > >> We already have a capability to capture bolt errors and > >>>>>> validation > >>>>>>>>> > errors > >>>>>>>>> > >> and pipe them into a Kafka topic. I want to propose that > >> we > >>>>>>> attach a > >>>>>>>>> > >> writer topology to the error and validation failed kafka > >>>>> topics > >>>>>> so > >>>>>>>>> > that we > >>>>>>>>> > >> can (a) create a new ES index for these errors and (b) > >>>>> create a > >>>>>>> new > >>>>>>>>> > Kibana > >>>>>>>>> > >> dashboard to visualize them. The benefit would be that > >>> errors > >>>>>> and > >>>>>>>>> > >> validation failures would be easier to see and analyze. > >>>>>>>>> > >> > >>>>>>>>> > >> I am seeking feedback on the following: > >>>>>>>>> > >> > >>>>>>>>> > >> - How granular would we want this feature to be? Think > > we > >>>>> would > >>>>>>> want > >>>>>>>>> > one > >>>>>>>>> > >> index/dashboard per source? Or would it be better to > >>> collapse > >>>>>>>>> > everything > >>>>>>>>> > >> into the same index? > >>>>>>>>> > >> - Do we care about storing these errors in HDFS as well? > >> Or > >>>>> is > >>>>>>>>> indexing > >>>>>>>>> > >> them enough? > >>>>>>>>> > >> - What types of errors should we record? I am proposing: > >>>>>>>>> > >> > >>>>>>>>> > >> For error reporting: > >>>>>>>>> > >> --Message failed to parse > >>>>>>>>> > >> --Enrichment failed to enrich > >>>>>>>>> > >> --Threat intel feed failures > >>>>>>>>> > >> --Generic catch-all for all other errors > >>>>>>>>> > >> > >>>>>>>>> > >> For validation reporting: > >>>>>>>>> > >> --What part of message failed validation > >>>>>>>>> > >> --What stellar validator caused the failure > >>>>>>>>> > >> > >>>>>>>>> > >> ------------------- > >>>>>>>>> > >> Thank you, > >>>>>>>>> > >> > >>>>>>>>> > >> James Sirota > >>>>>>>>> > >> PPMC- Apache Metron (Incubating) > >>>>>>>>> > >> jsirota AT apache DOT org > >>>>>>>>> > >> > >>>>>>>>> > >> -- > >>>>>>>>> > >> > >>>>>>>>> > >> Jon > >>>>>>>>> > >> > >>>>>>>>> > >> Sent from my mobile device > >>>>>>>>> > > > >>>>>>>>> > > ------------------- > >>>>>>>>> > > Thank you, > >>>>>>>>> > > > >>>>>>>>> > > James Sirota > >>>>>>>>> > > PPMC- Apache Metron (Incubating) > >>>>>>>>> > > jsirota AT apache DOT org > >>>>>>>>> > > > >>>>>>>>> > > -- > >>>>>>>>> > > > >>>>>>>>> > > Jon > >>>>>>>>> > > > >>>>>>>>> > > Sent from my mobile device > >>>>>>>>> > > >>>>>>>>> > ------------------- > >>>>>>>>> > Thank you, > >>>>>>>>> > > >>>>>>>>> > James Sirota > >>>>>>>>> > PPMC- Apache Metron (Incubating) > >>>>>>>>> > jsirota AT apache DOT org > >>>>>>>>> > > >>>>>>>>> -- > >>>>>>>>> > >>>>>>>>> Jon > >>>>>>>>> > >>>>>>>>> Sent from my mobile device > >>>>>>> ------------------- > >>>>>>> Thank you, > >>>>>>> > >>>>>>> James Sirota > >>>>>>> PPMC- Apache Metron (Incubating) > >>>>>>> jsirota AT apache DOT org > >>>>>>> > >>>>> -- > >>>>> > >>>>> Jon > >>>>> > >>>>> Sent from my mobile device > >>>>> > >>>> > >> -- > >> > >> Jon > >> > >> Sent from my mobile device > >> > > -- Jon Sent from my mobile device
