Hi, ListenSyslog (using the approach that is being discussed currently in another thread – ListenSyslog running on primary node as RGP, all other nodes connecting to the port that the RPG exposes). Various enrichment, routing on attributes etc. and finally into HDFS as Avro. I want to branch off at an appropriate point in the flow and do some further realtime analysis – got the output to port feeding to Spark process working fine (notwithstanding the issue that you have been so kind to help with previously with the SSLContext), just thinking about if this is most appropriate solution.
I have dabbled with a custom processor (for enriching url splitting/ enriching etc. – probably could have done with ExecuteScript processor in hindsight) so am comfortable with going this route if that is deemed more appropriate. Thanks Conrad From: Bryan Bende <bbe...@gmail.com> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> Date: Thursday, 2 June 2016 at 13:12 To: "users@nifi.apache.org" <users@nifi.apache.org> Subject: Re: Spark or custom processor? Conrad, I would think that you could do this all in NiFi. How do the log files come into NiFi? TailFile, ListenUDP/ListenTCP, List+FetchFile? -Bryan On Thu, Jun 2, 2016 at 6:41 AM, Conrad Crampton <conrad.cramp...@secdata.com<mailto:conrad.cramp...@secdata.com>> wrote: Hi, Any advice on ‘best’ architectural approach whereby some processing function has to be applied to every flow file in a dataflow with some (possible) output based on flowfile content. e.g. inspect log files for specific ip then send message to syslog approach 1 Spark Output port from NiFi -> Spark listens to that stream -> processes and outputs accordingly Advantages – scale spark job on Yarn, decoupled (reusable) from NiFi Disadvantages – adds complexity, decoupled from NiFi. Approach 2 NiFi Custom processor -> PutSyslog Advantages – reuse existing NiFi processors/ capability, obvious flow (design intent) Disadvantages – scale?? Any comments/ advice/ experience of either approaches? Thanks Conrad SecureData, combating cyber threats ________________________________ The information contained in this message or any of its attachments may be privileged and confidential and intended for the exclusive use of the intended recipient. If you are not the intended recipient any disclosure, reproduction, distribution or other dissemination or use of this communications is strictly prohibited. The views expressed in this email are those of the individual and not necessarily of SecureData Europe Ltd. Any prices quoted are only valid if followed up by a formal written quote. SecureData Europe Limited. Registered in England & Wales 04365896. Registered Address: SecureData House, Hermitage Court, Hermitage Lane, Maidstone, Kent, ME16 9NT ***This email originated outside SecureData*** Click here<https://www.mailcontrol.com/sr/0Yez0Z9rJiDGX2PQPOmvUr11KAWLA5a39FXrkhyyO4eQg2DXa9Xl!rwzg+4hlLPKdufvfzcRzpTaNxM9hG2QrA==> to report this email as spam.
