Hi Andrew, We tested your 0.0.6 binary and it seemed to do what we want. Unfortunately it is not compatible with the current Nifi version. Do you have a binary available that would work with nifi 1.0.0 ?
Thx. On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis <psaltis.and...@gmail.com> wrote: > Davy, > The processor I have been working on may meet your needs. You are correct, > at this time I have not pushed the source for it, still working through > some hurdles. The one thing to work out is how you would dynamically add > the processors -- suppose you may be able to use the REST API for NiFi. I > would imagine there are quite a number of these devices that you would need > to have processors for. In the use case I have been working on, there may > be 600 or so endpoints that I need to connect to and I'm trying to figure > out does it make sense to do it this way. > > I'll hopefully be in a place soon that I can push the code I have for the > GetTCP processor. > > > > > On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <bbe...@gmail.com> wrote: > >> Just wanted to clarify something about ListenTCP... it does support >> multiple incoming connections, however if you using the batch output >> capability, one flow file will contain data across all the connections. >> >> I do agree with Andrew that based on the description it sounds like NiFi >> is expected to be the client that initiates a connection and keeps reading >> for some amount of time/threshold, like a GetTCP processor. >> >> Currently we have ListenTCP which is waiting for incoming connections, >> and PutTCP which makes a connection, but only writes data. >> >> On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis <psaltis.and...@gmail.com >> > wrote: >> >>> Davy, >>> It sounds like you need a GetTCP type of processor that connects from >>> NiFi to the TCP endpoint on the sensor, is that correct? >>> >>> Thanks, >>> Andrew >>> >>> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <ddewa...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> Thanks for the response ... it's an existing network of sensors. The >>>> sensors spit out data over a serial interface that is exposed over a tcp >>>> connection. (rs232 -> ethernet converter in the sensor). >>>> The current sensor architecture involves clients making direct >>>> connections to the individual sensors. (establishing a tcp connection to >>>> the specific ip of the sensor). >>>> >>>> If I understand correctly, ListenTCP would not work in this case for >>>> multiple sensors >>>> >>>> Are you talking about a setup where the sensors would be in a "client" >>>> mode where each sensor would each establish a tcp connections to a single >>>> ListTCP processor ? >>>> >>>> Thx >>>> >>>> >>>> >>>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <joe.w...@gmail.com> wrote: >>>> >>>>> Hello >>>>> >>>>> Can you talk a bit about why you'd want ListenTCP processors tied to a >>>>> given sensor? You should be able to have many sensors to a single >>>>> ListenTCP. Each stream will be between a source/sensor and nifi so >>>>> data won't be getting intermingled there. If we're not providing >>>>> enough session/stream metadata on the flow files to make demux of the >>>>> streams easy using something like RouteOnAttribute or whatnot we >>>>> definitely should. >>>>> >>>>> Now, that said, you could certainly programmatically deploy (via the >>>>> REST API) instances of these processors along the lines of what your >>>>> endpoint registry tells you. It just seems on the surface like doing >>>>> so would be avoidable at least for the listening of data. Typically >>>>> such a registry would be useful to do additional tagging/enrichment of >>>>> the data and would occur once it is in the flow. >>>>> >>>>> Thanks >>>>> Joe >>>>> >>>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele <ddewa...@gmail.com> >>>>> wrote: >>>>> > We have a large number of sensors that send out data via TCP. The >>>>> idea is to >>>>> > use a ListenTCP processor in Nifi to capture the data, do some >>>>> filtering / >>>>> > basic transformation before sending it upstream into our stack. >>>>> > >>>>> > We can configure individual ListenTCP processors for each sensor, >>>>> and that >>>>> > works fine when the number of sensors is small, but once you hit a >>>>> larger >>>>> > number if becomes cumbersome and difficult to manage. >>>>> > >>>>> > We have an inventory of those sensors (exposed via a REST service >>>>> endpoint), >>>>> > containing the sensor tcp information like ip and port) >>>>> > >>>>> > Is there an easy way to create these ListenTCP processors on the fly >>>>> based >>>>> > on a REST endpoint or some other external configuration ? How would >>>>> that >>>>> > work ? >>>>> > >>>>> > Thx. >>>>> >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Andrew >>> >>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >>> >> >> > > > -- > Thanks, > Andrew > > Subscribe to my book: Streaming Data <http://manning.com/psaltis> > <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> > twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >
