Re: Example Load Profile

[Mark Payne]

James,

Sorry, I let this slip through the cracks. Just took a look at the template 
that you attached.

What you're doing here should certainly work okay with NiFi. However, the 
pattern that you are
using is to take a large XML file, split it into small chunks, process each 
chunk individually, and
then merge the small chunks back into a large chunk. This tends to be quite 
expensive and may
not yield the best performance.

NiFi would actually perform *far* better if you perform your processing without 
splitting that data
apart and then re-joining it. To that end, in NiFi 1.2.0 and 1.3.0 we have 
introduced several new
processors for handling record-oriented data [1] [2]. Unfortunately, we do not 
yet have a Record
Reader for XML data. However, there is a ScriptedRecordReader that could be 
used for scripting
one out, and I've seen one float around somewhere. Perhaps someone else on the 
list is able to
provide something there?

This approach, though, should make the flow much simpler to use and maintain. 
You'd be able to
unzip the data, then use QueryRecord to filter out the records you don't want, 
and then write the
results in JSON. So your flow would probably be as simple as:

GetFile -> UnpackContent -> QueryRecord -> PutFile

Thanks
-Mark

[1] https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi
[2] https://blogs.apache.org/nifi/entry/real-time-sql-on-event




On Jun 28, 2017, at 10:12 AM, Stephen Rathnam 
mailto:climbinggiant...@gmail.com>> wrote:

Hi everyone,

I am also starting to do some performance testing of a similar nature. Did
anyone find a good load profile/xml flow to use for this?

Regards,
Stephen

-
From: James Farrington mailto:nifini...@gmail.com>>
Date: Mon, Jun 19, 2017 at 11:26 AM
Subject: Example Load Profile
To: dev@nifi.apache.org


Hello All,

I am trying to get a performance benchmark for what size load my server can
handle. Currently we are running 3 nodes each with the following
configuration: 4 CPUs, 30.5GB RAM, and 80GB SSD.

Additionally, we are currently using an example XML file from online
(attached).

First off, would anyone have a good idea of a sample load profile that
would give a good baseline of how much my setup can handle? Secondly, would
anyone suggest a different XML file to use for a bench marking test (this
one was just found online).

In my search thus far I have not found any documentation on an example load
profile for a given configuration, so it would perhaps be helpful to others
in the future to have this formally documented. As well as adding a good
sample XML file for getting some benchmark data.

Any advice would be great!

Thanks,
James

Re: Example Load Profile

[Stephen Rathnam]

Hi everyone,

I am also starting to do some performance testing of a similar nature. Did
anyone find a good load profile/xml flow to use for this?

Regards,
Stephen

-
From: James Farrington 
Date: Mon, Jun 19, 2017 at 11:26 AM
Subject: Example Load Profile
To: dev@nifi.apache.org


Hello All,

I am trying to get a performance benchmark for what size load my server can
handle. Currently we are running 3 nodes each with the following
configuration: 4 CPUs, 30.5GB RAM, and 80GB SSD.

Additionally, we are currently using an example XML file from online
(attached).

First off, would anyone have a good idea of a sample load profile that
would give a good baseline of how much my setup can handle? Secondly, would
anyone suggest a different XML file to use for a bench marking test (this
one was just found online).

In my search thus far I have not found any documentation on an example load
profile for a given configuration, so it would perhaps be helpful to others
in the future to have this formally documented. As well as adding a good
sample XML file for getting some benchmark data.

Any advice would be great!

Thanks,
James