Re: ListenTCP not generating flowfiles

[Michael Loftis]

On Thu, Feb 10, 2022 at 4:00 PM Breno Cesar  wrote:
>
> Michael, Natan and Joe,
>
> Thank you very much for your replies.
> Here some stuffs that i done today:
>
> Changing the Batching Message Delimiter had no effect, according to the 
> documentation it only works when a package in TCP is already identified, and 
> apparently how is done and their delimiter is "hardcoded" in nifi.
> Searching a bit I found this project, i tested it here, and put "0" as the 
> delimiter of the income packages, and apparently it worked, i received some 
> connection flowfiles, I'll study the packages a little better and see which 
> delimiter (or sequence of delimiters) works best for my situation.

A fixed delimiter will not work correctly with IDPR.  It's a variable
length binary protocol, which means it'll need it's own RecordReader
to ingest correctly into NiFiandhonestly likely to do any real
work with it anyway...  So you either have to build a record reader,
or put something in front that'll digest the IDPR messages into
something NiFi can work with.

> I'm not a developer (I just know how to google well), and making an 
> IDPRRecordReader from scratch knowing the basics,  it might take a few years.
> i'm going to open a request in jira to develop the IDPRRecordReader, as well 
> to check if they could add the functionality to change the delimiter of 
> income packages in listentcp, and pray that someone cares about =p.
>
> Again guys, thank you very much for your help.
>
> Breno
>
>
>
> Em qui., 10 de fev. de 2022 às 00:13, Joe Witt  escreveu:
>>
>> Looking at
>> https://datatracker.ietf.org/doc/html/draft-ietf-idpr-specv1#section-2.4
>>
>> I think youll want to write an IDPRRecordReader
>>
>> Thanks
>>
>> On Wed, Feb 9, 2022 at 8:05 PM Nathan Gough  wrote:
>>>
>>> You should be able to set the message delimiter in ListenTCP with the 
>>> "Message Delimiter" property. The default is \n. I don't believe there's a 
>>> great way to see how data is being interpreted other than using interactive 
>>> debug mode with NiFi set through the bootstrap.conf.
>>>
>>> On Wed, Feb 9, 2022 at 9:29 PM Breno Cesar  wrote:

 Hi guys,

 First of all i apologize for my english, and thank you very much for your 
 answers.
 The data type i need is a data stream, which starts when the ipdr protocol 
 connect process ends.
 Is there any way to change the line terminator?  or somehow see how nifi 
 is interpreting the data when It its received ?

 Em qua., 9 de fev. de 2022 22:31, Joe Witt  escreveu:
>
> Correct.  With both ListenTCP and ListenTCPRecord it listens for data and 
> needs some way of framing them to generate records/bundles of data as 
> flowfiles.
>
> What kind of data do you want to listen for?
>
> Thanks
>
> On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis  wrote:
>>
>> ListenTCP isn’t fit for ingesting arbitrary data. It’s record oriented, 
>> default separator is Unix newline “\n”. I don’t think it will generate a 
>> flow if it never receives a newline before the connection closes.
>>
>> On Wed, Feb 9, 2022 at 18:17 Breno Cesar  
>> wrote:
>>>
>>> i'm tring to receive a tcp stream using the listentcp module, but when 
>>> the connection start, i see the traffic reaching my server (usgin 
>>> tcpdump), but the nifi does note generate a flow file.
>>> Ther is no error ow warning in the module or in the nifi cluster.
>>> Changing in the processor the bulleting level to "debug" , i can see 
>>> the following message for every package that i receive:
>>>
>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000--40efa66e] 
>>> Accepted incoming connection from /192.0.2.12:55209
>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000--40efa66e] 
>>> bytes read 46
>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000--40efa66e] No 
>>> more data available, returning for selection
>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000--40efa66e] 
>>> Reached EOF, closing connection
>>>
>>> bellow is the same message, but in the tcpdump format.
>>>
>>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>>> Transmission Control Protocol, Src Port:  55209  , Dst Port: 4737, Seq: 
>>> 1, Ack: 1, Len: 46
>>> Source Port:  55209
>>> Destination Port: 4737
>>> 
>>> 
>>> [Stream index: 0]
>>> [TCP Segment Len: 46]
>>> Sequence number: 1(relative sequence number)
>>> Sequence number (raw): 3765145522
>>> [Next sequence number: 47(relative sequence number)]
>>> Acknowledgment number: 1(relative ack number)
>>> Acknowledgment number (raw): 274707205
>>> 0101  = Header Length: 20 bytes (5)
>>> Flags: 0x018 (PSH, ACK)
>>> Window size value: 1
>>> [Calculated window 

Re: ListenTCP not generating flowfiles

[Breno Cesar]

Michael, Natan and Joe,

Thank you very much for your replies.
Here some stuffs that i done today:

   - Changing the Batching Message Delimiter had no effect, according to
   the documentation it only works when a package in TCP is already
   identified, and apparently how is done and their delimiter is "hardcoded"
   in nifi.
   - Searching a bit I found this project
   , i tested
   it here, and put "0" as the delimiter of the income packages, and
   apparently it worked, i received some connection flowfiles, I'll study the
   packages a little better and see which delimiter (or sequence of
   delimiters) works best for my situation.
   - I'm not a developer (I just know how to google well), and making an
   IDPRRecordReader from scratch knowing the basics,  it might take a few
   years.
   - i'm going to open a request in jira to develop the IDPRRecordReader,
   as well to check if they could add the functionality to change the
   delimiter of income packages in listentcp, and pray that someone cares
   about =p.

Again guys, thank you very much for your help.

*Breno *




Em qui., 10 de fev. de 2022 às 00:13, Joe Witt 
escreveu:

> Looking at
> https://datatracker.ietf.org/doc/html/draft-ietf-idpr-specv1#section-2.4
>
> I think youll want to write an IDPRRecordReader
>
> Thanks
>
> On Wed, Feb 9, 2022 at 8:05 PM Nathan Gough  wrote:
>
>> You should be able to set the message delimiter in ListenTCP with the
>> "Message Delimiter" property. The default is \n. I don't believe there's a
>> great way to see how data is being interpreted other than using interactive
>> debug mode with NiFi set through the bootstrap.conf.
>>
>> On Wed, Feb 9, 2022 at 9:29 PM Breno Cesar 
>> wrote:
>>
>>> Hi guys,
>>>
>>> First of all i apologize for my english, and thank you very much for
>>> your answers.
>>> The data type i need is a data stream, which starts when the ipdr
>>> protocol connect process ends.
>>> Is there any way to change the line terminator?  or somehow see how nifi
>>> is interpreting the data when It its received ?
>>>
>>> Em qua., 9 de fev. de 2022 22:31, Joe Witt 
>>> escreveu:
>>>
 Correct.  With both ListenTCP and ListenTCPRecord it listens for data
 and needs some way of framing them to generate records/bundles of data as
 flowfiles.

 What kind of data do you want to listen for?

 Thanks

 On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis 
 wrote:

> ListenTCP isn’t fit for ingesting arbitrary data. It’s record
> oriented, default separator is Unix newline “\n”. I don’t think it will
> generate a flow if it never receives a newline before the connection 
> closes.
>
> On Wed, Feb 9, 2022 at 18:17 Breno Cesar 
> wrote:
>
>> i'm tring to receive a tcp stream using the listentcp module, but
>> when the connection start, i see the traffic reaching my server (usgin
>> tcpdump), but the nifi does note generate a flow file.
>> Ther is no error ow warning in the module or in the nifi cluster.
>> Changing in the processor the bulleting level to "debug" , i can see
>> the following message for every package that i receive:
>>
>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000--40efa66e]
>> Accepted incoming connection from /192.0.2.12:55209
>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000--40efa66e]
>> bytes read 46
>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000--40efa66e]
>> No more data available, returning for selection
>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000--40efa66e]
>> Reached EOF, closing connection
>>
>> bellow is the same message, but in the tcpdump format.
>>
>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>> Transmission Control Protocol, Src Port:  55209  , Dst Port: 4737,
>> Seq: 1, Ack: 1, Len: 46
>> Source Port:  55209
>> Destination Port: 4737
>> 
>> 
>> [Stream index: 0]
>> [TCP Segment Len: 46]
>> Sequence number: 1(relative sequence number)
>> Sequence number (raw): 3765145522
>> [Next sequence number: 47(relative sequence number)]
>> Acknowledgment number: 1(relative ack number)
>> Acknowledgment number (raw): 274707205
>> 0101  = Header Length: 20 bytes (5)
>> Flags: 0x018 (PSH, ACK)
>> Window size value: 1
>> [Calculated window size: 1]
>> [Window size scaling factor: 1]
>> Checksum: 0x5fd7 [unverified]
>> [Checksum Status: Unverified]
>> Urgent pointer: 0
>> [SEQ/ACK analysis]
>> [Timestamps]
>> TCP payload (46 bytes)
>> [PDU Size: 46]
>> IPDR
>> Version: 2
>> Message id: CONNECT (5)
>> Session id: 0
>> Message flags: 0x00
>> Message length: 46
>>   

Re: Access to Community Slack

[Bryan Bende]

Hello,

We just updated the link on the site a few days ago.

Can you try the "invite" link here?

https://nifi.apache.org/mailing_lists.html

Thanks,

Bryan

On Thu, Feb 10, 2022 at 8:46 AM Nathaneal James  wrote:
>
> Hello,
>
> Is it possible to get an invite to the Apache Community Slack?
> I'm having trouble with Apache Nifi on Mac OSX.
>
>
> Thanks
> Nate

Access to Community Slack

[Nathaneal James]

Hello,

Is it possible to get an invite to the Apache Community Slack?
I'm having trouble with Apache Nifi on Mac OSX.


Thanks
Nate

Re: How to add a Content Viewer for Google Protobuf

[Chris Sampson]

My guess would be that you'd need to update the
StandardContentViewerController to expect and handle a new MIME type
relevant to your data.

As mentioned in the user mailing list for your previous query, you'll need
to figure a way of obtaining the schema for the data then convert it to a
more human-readable format to show in the browser. An example of that could
be to look at what is done for Avro [1], although the Avro schema is
typically included within the data itself.

I'm not an expert in this though, just took a quick look. Hope it helps.


[1]
https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-content-viewer/src/main/java/org/apache/nifi/web/StandardContentViewerController.java#L109



Cheers,

Chris Sampson

On Thu, 10 Feb 2022, 07:05 shweta julur,  wrote:

> Hi All,
>
> We are using Google Protobuf as an internal data format for a NiFi
> workflow. So we want to check the flow file contents after each processor
> has finished processing. As of now, there is no default content viewer for
> Google Protobuf. Is there any way to add one to be able to see the flow
> file contents?
>
> Any help would be appreciated.
>
> Thanks.
>