Good morning Davy, I have all (barring tests that fail) code in place to fulfill NIFI-2615 [1]. I plan on wrapping up the testing of it today and submitting a P/R. One thing to call out is that at present it is it's own module. However, I plan on adding it to the NiFi Standard Processor bundle before submitting the P/R. Let me know if you want me to add in the processing for the demarcator splitting.
Thanks, Andrew On Mon, Oct 31, 2016 at 4:37 PM, Andrew Psaltis <psaltis.and...@gmail.com> wrote: > Hi Davy, > Thanks for the P/R - I merged that in just a little bit ago. > > I think the feature you are describing makes total sense, I'm sure this > would be useful in many different use cases. I am in the process of > wrapping up the changes to fulfill JIRA NIFI-2615 [1]. Do you want to add > the "delimiter based spliter" feature to that JIRA? Are you planning on > implementing it or are you asking if I would? > > I should have the updated bits up in GH this evening/very early am. > > Thanks your kinda words regarding our loss. > > Thanks, > Andrew > > On Sun, Oct 30, 2016 at 7:26 PM, Davy De Waele <ddewa...@gmail.com> wrote: > >> Hi Andrew, >> >> I've looked at your code and submitted a pull request as I needed to do >> some code fixes to get it working >> Also posted an issue / question on how you see this GetTCP processor >> evolve. >> >> We're using Nifi to capture sensor data over TCP so a good GetTCP is >> definitely something we need in our Nifi tool-belt >> >> We would like to use it in a slightly different manner as per the current >> implementation. We would like to be able to specify a delimiter (byte >> sequence) in order to generate flow files out of this processor. >> Our sensor devices use a delimiter (end of transmission byte) to mark the >> end of a message in the tcp stream. >> >> That way, the GetTCP processor would already "split" the tcp stream into >> separate FlowFiles where each FlowFile would correspond to a response >> received from the tcp stream. I've seen something similar in the ListenTCP >> processor. >> >> Sorry to hear about your loss. My condolences. >> >> Regards, >> Davy >> >> >> >> On Tue, Oct 18, 2016 at 10:16 AM, Andrew Psaltis < >> psaltis.and...@gmail.com> wrote: >> >>> Davey, >>> Sorry for the delay in getting this done, I was away from the keyboard >>> for almost 1.5 weeks after a death in the family. I have moved the GetTCP >>> processor code to it's own bundle and separate repo [1] with NiFi 1.0.0 as >>> the target. I'm not sure if that is the long term home for it. However, it >>> does feel a little cleaner then mucking with the NiFi Standard Bundle or me >>> making the assumption it would be adopted as part of the standard bundle. >>> >>> I know of at least one bug in the processor that has been reported -- it >>> fails to reconnect if the connection is severed. I will be working on this, >>> however, I will not be able to get to it till Wednesday night or most >>> likely Thursday morning. >>> >>> Thanks again for your patience and interest. >>> >>> Andrew >>> >>> [1] https://github.com/apsaltis/nifi-gettcp-bundle >>> >>> On Mon, Oct 3, 2016 at 10:56 AM, Davy De Waele <ddewa...@gmail.com> >>> wrote: >>> >>>> Hi Andrew, >>>> >>>> We'd be happy to test and provide feedback. >>>> We have a use-case now with Nifi that we're in the process of >>>> implementing. >>>> >>>> If you have sources or binaries feel free to let me know..... >>>> >>>> Regards, >>>> Davy >>>> >>>> On Fri, Sep 30, 2016 at 9:24 PM, Andrew Psaltis < >>>> psaltis.and...@gmail.com> wrote: >>>> >>>>> Hi Davy, >>>>> Sorry for the slow response, I am traveling today. However, I should >>>>> be able to get a build of it for 1.0.0 out later today or tomorrow. >>>>> >>>>> Thanks, >>>>> Andrew >>>>> >>>>> On Fri, Sep 30, 2016 at 6:54 PM, Davy De Waele <ddewa...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Andrew, >>>>>> >>>>>> We tested your 0.0.6 binary and it seemed to do what we want. >>>>>> Unfortunately it is not compatible with the current Nifi version. >>>>>> Do you have a binary available that would work with nifi 1.0.0 ? >>>>>> >>>>>> Thx. >>>>>> >>>>>> On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis < >>>>>> psaltis.and...@gmail.com> wrote: >>>>>> >>>>>>> Davy, >>>>>>> The processor I have been working on may meet your needs. You are >>>>>>> correct, at this time I have not pushed the source for it, still working >>>>>>> through some hurdles. The one thing to work out is how you would >>>>>>> dynamically add the processors -- suppose you may be able to use the >>>>>>> REST >>>>>>> API for NiFi. I would imagine there are quite a number of these devices >>>>>>> that you would need to have processors for. In the use case I have been >>>>>>> working on, there may be 600 or so endpoints that I need to connect to >>>>>>> and >>>>>>> I'm trying to figure out does it make sense to do it this way. >>>>>>> >>>>>>> I'll hopefully be in a place soon that I can push the code I have >>>>>>> for the GetTCP processor. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 29, 2016 at 5:22 AM, Bryan Bende <bbe...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Just wanted to clarify something about ListenTCP... it does support >>>>>>>> multiple incoming connections, however if you using the batch output >>>>>>>> capability, one flow file will contain data across all the connections. >>>>>>>> >>>>>>>> I do agree with Andrew that based on the description it sounds like >>>>>>>> NiFi is expected to be the client that initiates a connection and keeps >>>>>>>> reading for some amount of time/threshold, like a GetTCP processor. >>>>>>>> >>>>>>>> Currently we have ListenTCP which is waiting for incoming >>>>>>>> connections, and PutTCP which makes a connection, but only writes data. >>>>>>>> >>>>>>>> On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis < >>>>>>>> psaltis.and...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Davy, >>>>>>>>> It sounds like you need a GetTCP type of processor that connects >>>>>>>>> from NiFi to the TCP endpoint on the sensor, is that correct? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Andrew >>>>>>>>> >>>>>>>>> On Thu, Sep 29, 2016 at 4:46 AM, Davy De Waele <ddewa...@gmail.com >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Thanks for the response ... it's an existing network of sensors. >>>>>>>>>> The sensors spit out data over a serial interface that is exposed >>>>>>>>>> over a >>>>>>>>>> tcp connection. (rs232 -> ethernet converter in the sensor). >>>>>>>>>> The current sensor architecture involves clients making direct >>>>>>>>>> connections to the individual sensors. (establishing a tcp >>>>>>>>>> connection to >>>>>>>>>> the specific ip of the sensor). >>>>>>>>>> >>>>>>>>>> If I understand correctly, ListenTCP would not work in this case >>>>>>>>>> for multiple sensors >>>>>>>>>> >>>>>>>>>> Are you talking about a setup where the sensors would be in a >>>>>>>>>> "client" mode where each sensor would each establish a tcp >>>>>>>>>> connections to a >>>>>>>>>> single ListTCP processor ? >>>>>>>>>> >>>>>>>>>> Thx >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Sep 28, 2016 at 10:03 PM, Joe Witt <joe.w...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hello >>>>>>>>>>> >>>>>>>>>>> Can you talk a bit about why you'd want ListenTCP processors >>>>>>>>>>> tied to a >>>>>>>>>>> given sensor? You should be able to have many sensors to a >>>>>>>>>>> single >>>>>>>>>>> ListenTCP. Each stream will be between a source/sensor and nifi >>>>>>>>>>> so >>>>>>>>>>> data won't be getting intermingled there. If we're not providing >>>>>>>>>>> enough session/stream metadata on the flow files to make demux >>>>>>>>>>> of the >>>>>>>>>>> streams easy using something like RouteOnAttribute or whatnot we >>>>>>>>>>> definitely should. >>>>>>>>>>> >>>>>>>>>>> Now, that said, you could certainly programmatically deploy (via >>>>>>>>>>> the >>>>>>>>>>> REST API) instances of these processors along the lines of what >>>>>>>>>>> your >>>>>>>>>>> endpoint registry tells you. It just seems on the surface like >>>>>>>>>>> doing >>>>>>>>>>> so would be avoidable at least for the listening of data. >>>>>>>>>>> Typically >>>>>>>>>>> such a registry would be useful to do additional >>>>>>>>>>> tagging/enrichment of >>>>>>>>>>> the data and would occur once it is in the flow. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> Joe >>>>>>>>>>> >>>>>>>>>>> On Wed, Sep 28, 2016 at 3:39 PM, Davy De Waele < >>>>>>>>>>> ddewa...@gmail.com> wrote: >>>>>>>>>>> > We have a large number of sensors that send out data via TCP. >>>>>>>>>>> The idea is to >>>>>>>>>>> > use a ListenTCP processor in Nifi to capture the data, do some >>>>>>>>>>> filtering / >>>>>>>>>>> > basic transformation before sending it upstream into our stack. >>>>>>>>>>> > >>>>>>>>>>> > We can configure individual ListenTCP processors for each >>>>>>>>>>> sensor, and that >>>>>>>>>>> > works fine when the number of sensors is small, but once you >>>>>>>>>>> hit a larger >>>>>>>>>>> > number if becomes cumbersome and difficult to manage. >>>>>>>>>>> > >>>>>>>>>>> > We have an inventory of those sensors (exposed via a REST >>>>>>>>>>> service endpoint), >>>>>>>>>>> > containing the sensor tcp information like ip and port) >>>>>>>>>>> > >>>>>>>>>>> > Is there an easy way to create these ListenTCP processors on >>>>>>>>>>> the fly based >>>>>>>>>>> > on a REST endpoint or some other external configuration ? How >>>>>>>>>>> would that >>>>>>>>>>> > work ? >>>>>>>>>>> > >>>>>>>>>>> > Thx. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks, >>>>>>>>> Andrew >>>>>>>>> >>>>>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>>>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>>>>>>> twiiter: @itmdata >>>>>>>>> <http://twitter.com/intent/user?screen_name=itmdata> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks, >>>>>>> Andrew >>>>>>> >>>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>>>>> twiiter: @itmdata >>>>>>> <http://twitter.com/intent/user?screen_name=itmdata> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Andrew >>>>> >>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >>>>> >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Andrew >>> >>> Subscribe to my book: Streaming Data <http://manning.com/psaltis> >>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> >>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> >>> >> >> > > > -- > Thanks, > Andrew > > Subscribe to my book: Streaming Data <http://manning.com/psaltis> > <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> > twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata> > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
