Corey, You could probably accomplish this currently with UpdateAttributes (it would be exhausting) to break out the flowfile content into attributes and then form them into JSON. I think a better option would be to use the ExecuteScript processor with something like netflow-parser [1] and process the incoming data into attributes until a custom processor is made available.
[1] https://github.com/rsvato/netflow-parser/blob/master/src/main/java/netflow/LineProcessor.java <https://github.com/rsvato/netflow-parser/blob/master/src/main/java/netflow/LineProcessor.java> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 18, 2017, at 9:35 AM, Corey Flowers <cflow...@onyxpoint.com> wrote: > > Thanks for all the help, I am going to look into Metron but I think to get > around my current problem of netflow sensor collection from multiple points, > I am going to use a listenUDP, with an execute stream running nfdump to > structure the data into a json format. I don't usually work with netflow data > but couldn't a processor be built that was something like > "ExtractNetFlowAttributes" Which would then load the netflow fields into > attributes and allow you to use something like attributestoJSON processor to > map to the proper structures? I am guessing you could even look in the file > to determine version or have a drop down to specify. > > Again, I am not a developer, nor do I use Netflow data enough to really know > the best course. > > Thanks for all the help! > You guys are awesome as always! > > Later! > > > > On Mon, Apr 17, 2017 at 7:28 PM, Andre <andre-li...@fucs.org > <mailto:andre-li...@fucs.org>> wrote: > Corey, > > I am currently assigned with a JIRA ticket to create this processor but I > have a few other tickets I am working one before I touch this one, so unless > someone has something to contribute I don't expect to get it ready in time > for 1.2. > > Cheers > > On Tue, Apr 18, 2017 at 12:57 AM, Corey Flowers <cflow...@onyxpoint.com > <mailto:cflow...@onyxpoint.com>> wrote: > Good morning everyone, > > Was there ever a netflow parser processor built? I am currently working > on a netflow issue and have seen several people discuss netflow parsing in > threads but I haven't seen any information about the release of a netflow > processor. The version I am currently working with is v9. > > Thanks! > > On Mon, Aug 8, 2016 at 10:00 AM, Madhukar Thota <madhukar.th...@gmail.com > <mailto:madhukar.th...@gmail.com>> wrote: > Hi Joe, > > we have bunch of cisco router that has netflow feature built in. Netflow > allows us to collect the network traffic from the devices and able to send > them to udp destination for processing. As the data comes in raw, we need to > parse the incoming data and do transformation and send to them hdfs, kafka or > elasticseach or some other destination for analytics. > > Today we are using logstash as netflow collector and able to do > transformation and write them into elasticsearch for visualization. As we are > moving all the logstash processing work to nifi, we want to move the netflow > parsing to nifi too. > > https://github.com/logstash-plugins/logstash-codec-netflow > <https://github.com/logstash-plugins/logstash-codec-netflow> > http://blogs.cisco.com/security/step-by-step-setup-of-elk-for-netflow-analytics > > <http://blogs.cisco.com/security/step-by-step-setup-of-elk-for-netflow-analytics> > https://streamsets.com/documentation/datacollector/latest/help/#Origins/UDP.html#concept_rst_2y5_1s > > <https://streamsets.com/documentation/datacollector/latest/help/#Origins/UDP.html%23concept_rst_2y5_1s> > > > -Madhu > > On Mon, Aug 8, 2016 at 9:42 AM, Joe Witt <joe.w...@gmail.com > <mailto:joe.w...@gmail.com>> wrote: > Hello > > There are no processors included in the apache release that > specifically operate on netflow so you'd need to have a custom > processor to deal with it until one is included. > > Netflow is often flowing through NiFi typically for things like > content merging and loading into HDFS. Parsing is a good use case and > presumably after that you'd want to make some routing decisions or do > some sort of enrichment? Can you describe in more detail what you'd > like to be able to accomplish in NiFi and what systems it would > deliver the netflow to? Also, what type of Netflow is of interest (it > can be frustratingly proprietary)? > > Thanks > Joe > > On Mon, Aug 8, 2016 at 9:27 AM, Madhukar Thota <madhukar.th...@gmail.com > <mailto:madhukar.th...@gmail.com>> wrote: > > Is there any Processor available for Netflow? If not what is the best way to > > get Netflow data parsed using nifi? > > > > > > > > > -- > Corey Flowers > Vice President, Onyx Point, Inc > (410) 541-6699 <tel:(410)%20541-6699> > cflow...@onyxpoint.com <mailto:cflow...@onyxpoint.com> > > -- This account not approved for unencrypted proprietary information -- > > > > > -- > Corey Flowers > Vice President, Onyx Point, Inc > (410) 541-6699 > cflow...@onyxpoint.com <mailto:cflow...@onyxpoint.com> > > -- This account not approved for unencrypted proprietary information --
signature.asc
Description: Message signed with OpenPGP using GPGMail
