Re: [EXT] High-performance Nifi decoder

[Ali Nazemian]

Hi Peter,

It hasn't been finalized yet, but we can configure it to be reasonably low.
Basically the higher batch size. In regards to my question, which strategy
you think will provide more throughput?

Regards,
Ali

On Wed, Aug 23, 2017 at 6:49 PM, Peter Wicks (pwicks) 
wrote:

> Hi Ali,
>
>
>
> How many FlowFile’s are you expecting per minute? My experience with NiFi
> has been that if you have 1 FlowFile that is several GB’s in size you can
> frequently process it faster than 100k FlowFile’s at a fraction the size.
> There is a lot of overhead in managing the life cycle of a FlowFile when
> you start to have lots of them.
>
>
>
> --Peter
>
>
>
> *From:* Ali Nazemian [mailto:alinazem...@gmail.com]
> *Sent:* Wednesday, August 23, 2017 2:49 PM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] High-performance Nifi decoder
>
>
>
> Hi all,
>
>
>
>
>
> I am investigating the right approach for implementing an high-performance
> network packet decoder in Nifi. I am trying to achieve about 1 Gbps
> throughput on a specific decoder. (with 4 Nifi instances) I was wondering
> which option would perform better. 1) Implementing my decoder in a separate
> Nifi processor using NAR mechanism; 2) Implementing my decoder in C++ and
> use "ExecuteStreamCommand" to run that decoder. Basically, if
> ExecuteStreamCommand involves Disk for the sake of sending messages to the
> written script, it doesn't matter how fast the decoder is, disk would be
> the main bottleneck.
>
>
>
>
>
> Regards,
>
> Ali
>



-- 
A.Nazemian

RE: [EXT] High-performance Nifi decoder

[Peter Wicks (pwicks)]

Hi Ali,

How many FlowFile’s are you expecting per minute? My experience with NiFi has 
been that if you have 1 FlowFile that is several GB’s in size you can 
frequently process it faster than 100k FlowFile’s at a fraction the size. There 
is a lot of overhead in managing the life cycle of a FlowFile when you start to 
have lots of them.

--Peter

From: Ali Nazemian [mailto:alinazem...@gmail.com]
Sent: Wednesday, August 23, 2017 2:49 PM
To: users@nifi.apache.org
Subject: [EXT] High-performance Nifi decoder

Hi all,


I am investigating the right approach for implementing an high-performance 
network packet decoder in Nifi. I am trying to achieve about 1 Gbps throughput 
on a specific decoder. (with 4 Nifi instances) I was wondering which option 
would perform better. 1) Implementing my decoder in a separate Nifi processor 
using NAR mechanism; 2) Implementing my decoder in C++ and use 
"ExecuteStreamCommand" to run that decoder. Basically, if ExecuteStreamCommand 
involves Disk for the sake of sending messages to the written script, it 
doesn't matter how fast the decoder is, disk would be the main bottleneck.


Regards,
Ali