To add on to Lee’s excellent points, if you are concerned about capturing the total lifetime, you may be able to intercept the messages coming from the console application and insert an attribute for timing before Kafka processes those messages. You can then evaluate against that in NiFi. This will include the Kafka batching time, as the provenance metrics are NiFi-only.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On May 27, 2016, at 2:43 PM, Lee Laim <lee.l...@gmail.com> wrote:
>
> Igor,
>
> Here are 3 methods which may help:
>
>
> 1, Processor Statistics - 'Average Task Duration' is one of the processor
> statistics you can use to determine, on average, how fast an individual
> process takes. Additionally, 'Average Lineage duration' might give you some
> insight across multiple processors and queue times. (right click on the
> processor)
>
> 2. The Data Provenance feature has a graphical representation of the
> progression of the data flow, with an interactive slider bar.
> (4th icon on the upper right side of the browser)
>
> 3. You could also use two Update Attribute processors to make a 'elapsed'
> attribute to profile a specific portion of your data flow:
>
> start_time = ${now():toNumber()}
>
> data flow of some duration
>
> elapsed_time = ${now():toNumber():minus(${start_time})}
>
>
> -Lee
>
>
>
>
> On Fri, May 27, 2016 at 2:21 PM, Igor Kravzov <igork.ine...@gmail.com
> <mailto:igork.ine...@gmail.com>> wrote:
> It's probably a strange requirement.
> I want to measure how long will take for Kafka/NiFi to process a certain
> amount of messages.
> From one pint there is a console application which will send messages to
> Kafka in a loop.
> GetKafka will read messages in batches and Splittext will split these.
> Basically I want to see how long it took NiFi to process messages from Kafka.
> Any idea how to achieve this?
>
> Thanks in advance.
>
signature.asc
Description: Message signed with OpenPGP using GPGMail
