Joining two or more flow files and merging the content

[nathan.english]

Hi All,

I've got a case where I need to join two or more separate flow files, based on 
one or more 's ID within the flow (Multi Record flow file, each record with a 
unique ID). The content of the second flow file could also contain one or more 
records each again with a matching ID to the records in the first flow file. 
The second flow file comes from a different data source to the first and could 
be made to one record per flow file with the ID as an attribute if this makes 
it simpler.

It can take some time for there to be a match, so it will need to be able to 
queue and replay for a configurable period. If after a set period, there is no 
match release it downstream. When a match is found, the matching record in the 
secondary flow file needs to be injected into the content of the first flow 
file record.

I see there is the Lookup Record Processor, but there isn't a lookup service to 
query flow files in a queue or provenance? I want to avoid writing the 
secondary flow file to a database, or other storage option and would like to do 
the merge in flight through NiFi.

Below is a rough sketch of what I am trying to do.

Any suggestions would be much appreciated.

Kind Regards,

Nathan

[cid:image002.png@01D6CE1A.451A9590]

Re: 1 minutes vs 5 minutes Metrics via Prometheus

[jmkofoed]

Hi

Now I have checked all metrics and the following metrics have values going
from 0 up to X and back to 0 within 3 scrapings:
nifi_amount_items_queued
nifi_amount_threads_active
nifi_percent_used_bytes
nifi_percent_used_count
nifi_size_content_queued_total
nifi_transmitting

The following metrics have been checked with flows with 1-2 files per 10
minutes. These shows the same values for 5 minutes however if there are many
files they show changes at every scrap (15s). but still one file is
effecting the value for 5 minutes.
nifi_amount_bytes_read
nifi_amount_bytes_received
nifi_amount_bytes_sent
nifi_amount_bytes_transferred
nifi_amount_bytes_written
nifi_amount_flowfiles_received
nifi_amount_flowfiles_removed
nifi_amount_flowfiles_sent
nifi_amount_flowfiles_transferred
nifi_amount_items_input
nifi_amount_items_output
nifi_average_lineage_duration
nifi_size_content_input_total
nifi_size_content_output_total
nifi_total_bytes_read
nifi_total_bytes_written

kind regards
Jens M. Kofoed



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: 1 minutes vs 5 minutes Metrics via Prometheus

[jmkofoed]

Hi again,

This is a bit more complex because I see some metrics change every time
Prometheus is scraping (15 sec.)
The following metrics is updated frequently:
nifi_size_content_queued_total
nifi_amount_items_queued
nifi_amount_items_output
nifi_amount_bytes_received

But if the rate of flowfiles is very low (1 every 5-10 minutes)
nifi_size_content_input_total is showing the same value for 5 minutes. The
value change then new files arrived to the process (about every 1 minutes)
but the values looks like it comes from the 5 minutes stat. Not like the
metrics above.

kind regards
Jens M. Kofoed



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

1 minutes vs 5 minutes Metrics via Prometheus

[Jens M. Kofoed]

We are looking into getting metrics from NIFI into grafana via Prometheus.
It seems the metrics reported are the 5 minutes stats and not the 1 minutes
snap. If we look at the status history of a processor we can see that the
history is shown in a 1 minutes interval. I know the stat on the processor
is showing the last 5 minutes and it seems like it is the same 5
minutes stat that is being reported at the metrics instead of the 1 minutes
snap.

Is it possible to configure the Prometheus reporting task to report the 1
minutes snap instead?

kind regards
Jens M. Kofoed

NiFi Missing Provenance Data

[Leon Yu]

Hello,

This is the first time I’ve emailed here so please bear with me.

We are using nifi-1.12.1-RC2 and for one of our tests, we are seeing missing 
data provenance in NiFi despite the DB showing the flow being processed.
A simplified explanation of how the system works is NiFi ingests a text file, 
creates a flow file from text file, and some data is written to the DB.
The tests I was running ran two files through NiFi about 20 minutes apart.  The 
DB has shown both files having successfully been processed.  No other process 
writes to the DB other than NiFi.  NiFi’s data provenance only showed data for 
the 1st file, there was no trace of the 2nd file despite success flow.

Thank you,


[A close up of a sign  Description automatically generated]
  Leon Yu
  SDET, Macrohealth

   [A close up of a sign  Description automatically generated] 
[A picture containing drawing  Description 
automatically generated] 

Utilizing NetflowV9 in NiFi v1.12.1

[Thane Durey]

Hello all,

Has anyone found a solution to utilize NetflowV9? I know NiFi v1.12.1 has a
processor to parse Netflow version 5, but I haven't found any information
on Netflow version 9.

>From all the browsing I've done, I've only found a few tickets/issues that
discuss it in the issues.apache.org/jira page. But I don't know if it's
actively being worked on, or when it will be released

Has anyone implemented anything v9?

Any help would be appreciated,

Thanks,
T