Hi Andrew, I've looked at your code and submitted a pull request as I needed to do some code fixes to get it working Also posted an issue / question on how you see this GetTCP processor evolve.
We're using Nifi to capture sensor data over TCP so a good GetTCP is definitely something we need in our Nifi tool-belt We would like to use it in a slightly different manner as per the current implementation. We would like to be able to specify a delimiter (byte sequence) in order to generate flow files out of this processor. Our sensor devices use a delimiter (end of transmission byte) to mark the end of a message in the tcp stream. That way, the GetTCP processor would already "split" the tcp stream into separate FlowFiles where each FlowFile would correspond to a response received from the tcp stream. I've seen something similar in the ListenTCP processor. Sorry to hear about your loss. My condolences. Regards, Davy On Tue, Oct 18, 2016 at 10:16 AM, Andrew Psaltis <psaltis.and...@gmail.com> wrote: > Davey, > Sorry for the delay in getting this done, I was away from the keyboard for > almost 1.5 weeks after a death in the family. I have moved the GetTCP > processor code to it's own bundle and separate repo [1] with NiFi 1.0.0 as > the target. I'm not sure if that is the long term home for it. However, it > does feel a little cleaner then mucking with the NiFi Standard Bundle or me > making the assumption it would be adopted as part of the standard bundle. > > I know of at least one bug in the processor that has been reported -- it > fails to reconnect if the connection is severed. I will be working on this, > however, I will not be able to get to it till Wednesday night or most > likely Thursday morning. > > Thanks again for your patience and interest. > > Andrew > > [1] https://github.com/apsaltis/nifi-gettcp-bundle > > On Mon, Oct 3, 2016 at 10:56 AM, Davy De Waele <ddewa...@gmail.com> wrote: > >> Hi Andrew, >> >> We'd be happy to test and provide feedback. >> We have a use-case now with Nifi that we're in the process of >> implementing. >> >> If you have sources or binaries feel free to let me know..... >> >> Regards, >> Davy >> >> On Fri, Sep 30, 2016 at 9:24 PM, Andrew Psaltis <psaltis.and...@gmail.com >> > wrote: >> >>> Hi Davy, >>> Sorry for the slow response, I am traveling today. However, I should be >>> able to get a build of it for 1.0.0 out later today or tomorrow. >>> >>> Thanks, >>> Andrew >>> >>> On Fri, Sep 30, 2016 at 6:54 PM, Davy De Waele <ddewa...@gmail.com> >>> wrote: >>> >>>> Hi Andrew, >>>> >>>> We tested your 0.0.6 binary and it seemed to do what we want. >>>> Unfortunately it is not compatible with the current Nifi version. >>>> Do you have a binary available that would work with nifi 1.0.0 ? >>>> >>>> Thx. >>>> >>>> On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis < >>>> psaltis.and...@gmail.com> wrote: >>>> >>>>> Davy, >>>>> The processor I have been working on may meet your needs. You are >>>>> correct, at this time I have not pushed the source for it, still working >>>>> through some hurdles. The one thing to work out is how you would >>>>> dynamically add the processors -- suppose you may be able to use the REST >>>>> API for NiFi. I would imagine there are quite a number of these devices >>>>> that you would need to have processors for. In the use case I have been >>>>> working on, there may be 600 or so endpoints that I need to connect to and >>>>> I'm trying to figure out does it make sense to do it this way. >>>>> >>>>> I'll hopefully be in a place soon that I can push the code I have for >>>>> the GetTCP processor. >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <bbe...@gmail.com> wrote: >>>>> >>>>>> Just wanted to clarify something about ListenTCP... it does support >>>>>> multiple incoming connections, however if you using the batch output >>>>>> capability, one flow file will contain data across all the connections. >>>>>> >>>>>> I do agree with Andrew that based on the description it sounds like >>>>>> NiFi is expected to be the client that initiates a connection and keeps >>>>>> reading for some amount of time/threshold, like a GetTCP processor. >>>>>> >>>>>> Currently we have ListenTCP which is waiting for incoming >>>>>> connections, and PutTCP which makes a connection, but only writes data. >>>>>> >>>>>> On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis < >>>>>> psaltis.and...@gmail.com> wrote: >>>>>> >>>>>>> Davy, >>>>>>> It sounds like you need a GetTCP type of processor that connects >>>>>>> from NiFi to the TCP endpoint on the sensor, is that correct? >>>>>>> >>>>>>> Thanks, >>>>>>> Andrew >>>>>>> >>>>>>> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <ddewa...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Thanks for the response ... it's an existing network of sensors. >>>>>>>> The sensors spit out data over a serial interface that is exposed over >>>>>>>> a >>>>>>>> tcp connection. (rs232 -> ethernet converter in the sensor). >>>>>>>> The current sensor architecture involves clients making direct >>>>>>>> connections to the individual sensors. (establishing a tcp connection >>>>>>>> to >>>>>>>> the specific ip of the sensor). >>>>>>>> >>>>>>>> If I understand correctly, ListenTCP would not work in this case >>>>>>>> for multiple sensors >>>>>>>> >>>>>>>> Are you talking about a setup where the sensors would be in a >>>>>>>> "client" mode where each sensor would each establish a tcp connections >>>>>>>> to a >>>>>>>> single ListTCP processor ? >>>>>>>> >>>>>>>> Thx >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <joe.w...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello >>>>>>>>> >>>>>>>>> Can you talk a bit about why you'd want ListenTCP processors tied >>>>>>>>> to a >>>>>>>>> given sensor? You should be able to have many sensors to a single >>>>>>>>> ListenTCP. Each stream will be between a source/sensor and nifi so >>>>>>>>> data won't be getting intermingled there. If we're not providing >>>>>>>>> enough session/stream metadata on the flow files to make demux of >>>>>>>>> the >>>>>>>>> streams easy using something like RouteOnAttribute or whatnot we >>>>>>>>> definitely should. >>>>>>>>> >>>>>>>>> Now, that said, you could certainly programmatically deploy (via >>>>>>>>> the >>>>>>>>> REST API) instances of these processors along the lines of what >>>>>>>>> your >>>>>>>>> endpoint registry tells you. It just seems on the surface like >>>>>>>>> doing >>>>>>>>> so would be avoidable at least for the listening of data. >>>>>>>>> Typically >>>>>>>>> such a registry would be useful to do additional >>>>>>>>> tagging/enrichment of >>>>>>>>> the data and would occur once it is in the flow. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Joe >>>>>>>>> >>>>>>>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele <ddewa...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> > We have a large number of sensors that send out data via TCP. >>>>>>>>> The idea is to >>>>>>>>> > use a ListenTCP processor in Nifi to capture the data, do some >>>>>>>>> filtering / >>>>>>>>> > basic transformation before sending it upstream into our stack. >>>>>>>>> > >>>>>>>>> > We can configure individual ListenTCP processors for each >>>>>>>>> sensor, and that >>>>>>>>> > works fine when the number of sensors is small, but once you hit >>>>>>>>> a larger >>>>>>>>> > number if becomes cumbersome and difficult to manage. >>>>>>>>> > >>>>>>>>> > We have an inventory of those sensors (exposed via a REST >>>>>>>>> service endpoint), >>>>>>>>> > containing the sensor tcp information like ip and port) >>>>>>>>> > >>>>>>>>> > Is there an easy way to create these ListenTCP processors on the >>>>>>>>> fly based >>>>>>>>> > on a REST endpoint or some other external configuration ? How >>>>>>>>> would that >>>>>>>>> > work ? >>>>>>>>> > >>>>>>>>> > Thx. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks, >>>>>>> Andrew >>>>>>> >>>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>>>>> twiiter: @itmdata >>>>>>> <http://twitter.com/intent/user?screen_name=itmdata> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Andrew >>>>> >>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >>>>> >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Andrew >>> >>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >>> >> >> > > > -- > Thanks, > Andrew > > Subscribe to my book: Streaming Data <http://manning.com/psaltis> > <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> > twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >
