Davy, The processor I have been working on may meet your needs. You are correct, at this time I have not pushed the source for it, still working through some hurdles. The one thing to work out is how you would dynamically add the processors -- suppose you may be able to use the REST API for NiFi. I would imagine there are quite a number of these devices that you would need to have processors for. In the use case I have been working on, there may be 600 or so endpoints that I need to connect to and I'm trying to figure out does it make sense to do it this way.
I'll hopefully be in a place soon that I can push the code I have for the GetTCP processor. On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <bbe...@gmail.com> wrote: > Just wanted to clarify something about ListenTCP... it does support > multiple incoming connections, however if you using the batch output > capability, one flow file will contain data across all the connections. > > I do agree with Andrew that based on the description it sounds like NiFi > is expected to be the client that initiates a connection and keeps reading > for some amount of time/threshold, like a GetTCP processor. > > Currently we have ListenTCP which is waiting for incoming connections, and > PutTCP which makes a connection, but only writes data. > > On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis <psaltis.and...@gmail.com> > wrote: > >> Davy, >> It sounds like you need a GetTCP type of processor that connects from >> NiFi to the TCP endpoint on the sensor, is that correct? >> >> Thanks, >> Andrew >> >> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <ddewa...@gmail.com> >> wrote: >> >>> Hi, >>> >>> Thanks for the response ... it's an existing network of sensors. The >>> sensors spit out data over a serial interface that is exposed over a tcp >>> connection. (rs232 -> ethernet converter in the sensor). >>> The current sensor architecture involves clients making direct >>> connections to the individual sensors. (establishing a tcp connection to >>> the specific ip of the sensor). >>> >>> If I understand correctly, ListenTCP would not work in this case for >>> multiple sensors >>> >>> Are you talking about a setup where the sensors would be in a "client" >>> mode where each sensor would each establish a tcp connections to a single >>> ListTCP processor ? >>> >>> Thx >>> >>> >>> >>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <joe.w...@gmail.com> wrote: >>> >>>> Hello >>>> >>>> Can you talk a bit about why you'd want ListenTCP processors tied to a >>>> given sensor? You should be able to have many sensors to a single >>>> ListenTCP. Each stream will be between a source/sensor and nifi so >>>> data won't be getting intermingled there. If we're not providing >>>> enough session/stream metadata on the flow files to make demux of the >>>> streams easy using something like RouteOnAttribute or whatnot we >>>> definitely should. >>>> >>>> Now, that said, you could certainly programmatically deploy (via the >>>> REST API) instances of these processors along the lines of what your >>>> endpoint registry tells you. It just seems on the surface like doing >>>> so would be avoidable at least for the listening of data. Typically >>>> such a registry would be useful to do additional tagging/enrichment of >>>> the data and would occur once it is in the flow. >>>> >>>> Thanks >>>> Joe >>>> >>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele <ddewa...@gmail.com> >>>> wrote: >>>> > We have a large number of sensors that send out data via TCP. The >>>> idea is to >>>> > use a ListenTCP processor in Nifi to capture the data, do some >>>> filtering / >>>> > basic transformation before sending it upstream into our stack. >>>> > >>>> > We can configure individual ListenTCP processors for each sensor, and >>>> that >>>> > works fine when the number of sensors is small, but once you hit a >>>> larger >>>> > number if becomes cumbersome and difficult to manage. >>>> > >>>> > We have an inventory of those sensors (exposed via a REST service >>>> endpoint), >>>> > containing the sensor tcp information like ip and port) >>>> > >>>> > Is there an easy way to create these ListenTCP processors on the fly >>>> based >>>> > on a REST endpoint or some other external configuration ? How would >>>> that >>>> > work ? >>>> > >>>> > Thx. >>>> >>> >>> >> >> >> -- >> Thanks, >> Andrew >> >> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >> > > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
