[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 @ottobackwards @MikeThomsen Thanks for all your support and guidance in my first OS contribution. Soon, I will try to contribute for NetFlowv9 processor. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2820 @PrashanthVenkatesan if any of the builds passed, don't worry about it. It's the same test, just localized to different locale settings. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 You can trigger a build by closing and reopening the pr. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 @MikeThomsen One of the build failed due to some other reason. How can i retrigger the build without making new commit? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 Sorry for the delay @MikeThomsen . I will make changes and push by EOD. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2820 @PrashanthVenkatesan any updates? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2820 @PrashanthVenkatesan you also have a merge conflict. That'll need to be resolved as well. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2820 @joewitt @bbende AFAIK, I have no resources for testing this against a live Cisco system. What are your thoughts on merging it if everything checks out but I can't do a live test? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 Committers, Any one available to review this PR? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 @ottobackwards Thanks for your support. @MikeThomsen @bbende @mattyb149 - Can anyone review this if you bandwidth? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 As far as I can see, this PR is in great same. Good works @PrashanthVenkatesan. Tests and contrib tests are great. I manually tested with the sample generator and the output looks like it will be useful. Also, the factoring of the submittal will make the record reader follow up a breeze ;) +1 from me. @MikeThomsen, @bbende, @mattyb149 ? anyone have time to take a look at this? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 Also, thinking of it, the name of the Nar you have should be network processors nar, not just network nar. When we do the controller services (record readers) they will need a nar and that is the convention. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 You're right. I will format those fields**(srcaddr, dstaddr & nexthop)** as IP address.. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 OK. Contrib check and build passes. Everything runs the way it should. My question is on the data. The src and dest addresses are just the numbers. They are supposed to be ip addresses. I think that formatting them as IP addresses rather than just as numbers would be more usable. Is there something I'm missing? I don't have an example of something else that shows net flow data. Are there any other fields that are output 'raw' that could be formatted? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 I think conclusion at this point, 1. Sending the RAW binary message through "original" relationship 2. Remove the "template" relationship and document the json schema in additionalDetails.html .. I will do these changes and push the code. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 @bbende when he says Template he means a json record with schema information ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2820 I haven't looked at any of the code so keep that in mind :) If we are talking about NiFi templates, then they are just examples of how to use a processor or set of processors, which you can then give to someone to help them get started. We have a wiki page where people have posted some for the community: https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates I think the recent work @ottobackwards did with the syslog record readers is a good example of what we can do for this case. Meaning later on we can implement a NetflowV5RecordReader, which then lets you use stuff like ConvertRecord to go from netflow to any format like JSON, CSV, Avro. For now, if this processor always produces JSON, then documenting the expected output format in additionalDetails.html seems sufficient to me. I don't really know enough about what people do with netflow data to know if keeping the raw message with the parsed one makes sense. I would say typically we wouldn't keep the raw message with the parsed one, but you could always have an option to control that if you thought it was necessary. The processor can also have an original relationship as Otto suggested, although at that point the original and parsed data are completely separate. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 In Nifi, they way that the template would be done, is not through a Processor, but instead you would implement this functionality as a RecordReader with a set schema ( like the syslog schemas ). Then you could choose to write the data with the schema included in the avro. Sending a template is not something that processors in standard nifi usually do, it will be enough to just document the schema in the additionalDetails.html documentation. When this processor lands, I think I'm going to do the RecordReader. Then again, we should get someone else's opinion. @bbende can you chime in here? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 @ottobackwards So you mean to create "original" relationship and emit the binary RAW data to that relationship?? Fine I can do that change. Reg template, **netflowv5** template doesn't change. But going forward if we write processor for other related protocol like netflowv9, ipfix,etc.. These protocols have dynamic template(template will change at runtime, we need to parse that from the incoming packet)... ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 I think have an original relationship for flows that want to pass the data on as-is is a better option myself. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 Is that template _ever_ going to change? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 @ottobackwards You're right. Adding RAW to json is configurable property. I also see no usage of the RAW data downstream. I will remove that property.. Fine?? However templates is required, one scenario is "assume user want to convert this json to avro in downstream(external to nifi). That time he needs to know the **data type** of each field to create avro schema.. Sending ONE-TIME template , would be helpful in downstream to create schema." ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 @PrashanthVenkatesan It looked to me that the raw data was NOT part of the json. Am I mistaken? I don't think that is good practice, having json + something. If you put the raw content in a json field ( maybe encoded base64 or something ) that would work I think. So IF I configured output to content and IF I select include raw in output THEN encode the raw and put it in the json. Something like that. I don't understand how the output of the Netflow could be different where you would need templates. When would there be different json? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 @ottobackwards Thanks for your valuable review. I thought if user flow sends this json to some sink (say kafka), it would be good to have the raw_data with it. are you suggesting me to remove this ? .. Providing template also for the above purpose, if the external system wants to parse the raw or understand the parsing configs, template would be necessary. Also further ahead if we support other netflow related protocols that has concept of dynamic templates, it is worth sending templates via relationship. I understand your point adding raw to json looks like conflating, but i felt creating the another relationship would create separate content claim for each record that might degrade the throughput of processor. would like to know your view on my points.. I will incorporate all the other review comments and soon push the commit. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 Build with tests and contrib-check was fine as well ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 When this PR goes in, I'll create a jira for Netflow5RecordReader ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 I will try. Can you fill out the checkboxes in the PR template above? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 @ottobackwards Have any bandwidth to test this PR? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 @PrashanthVenkatesan thanks for putting all the work in on this, I'm not going to be able to test this for a couple of day most likely, but I will ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 [Generator.zip](https://github.com/apache/nifi/files/2177116/Generator.zip) @ottobackwards Attached data generator jar. Although this is rough code, hope it serves the purpose. Please check the readme for usage. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 Currently I can find only Windows applications. I'll check for any generator available that's runs in Linux. If not I will create a generator code and share it asap.. ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 Anything not windows specific? ---
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on the issue: https://github.com/apache/nifi/pull/2820 Flow Generator: https://flowalyzer-netflow-generator.soft112.com/ Flow Template: [NetFlowv5_Test_Template.zip](https://github.com/apache/nifi/files/2169500/NetFlowv5_Test_Template.zip) [This template receives data over UDP and use this custom processor to parse it] https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html - For more details ,Refer NetflowV5 section in this article. Hope these will be useful. ---
