Also, I just realized I misread your sampling requirement. You would use
the approach above if you wanted to sample *every 1032th flowfile*, but you
want a sample size of 1032 total. You can still use a randomizing
selection approach as I described (though your mod value would depend on
what frequency you want to draw samples), but you would also want to stop
sampling after 1032 flowfiles.
To do this, you can add an UpdateAttribute with "Store State Locally", and
a dynamic property named "counter" with a value of
"${getStateValue("counter"):plus(1)}" and a Stateful Variables Initial
Value of 0. Then add a RouteOnAttribute with a property called "samples"
with value "${counter:toNumber():le(1032)}}, and only the first 1032
flowfiles will be routed to that relationship.
On Thu, May 19, 2022 at 6:49 AM Joe Gresock <[email protected]> wrote:
> James,
>
> This sounds like an interesting project. I would recommend
> RouteOnAttribute with a "sample" property with value
> "${random():mod(1032):equals(100)}" (the second number could be anything
> between 0 and 1031), and then routing the "sample" relationship to your
> sampling path. I'm not sure I understand the stratified date aspect, but
> you may be able to do some routing there as well if only certain dates
> should be sampled.
>
> Hope this helps,
> Joe
>
> On Thu, May 19, 2022 at 6:20 AM James McMahon <[email protected]>
> wrote:
>
>> I have been tasked to draw samples from very large raw data sets for
>> triage analysis. I am to provide multiple sampling methods. Drawing a
>> random sample of N records is one method. A second method is to draw a
>> fixed sample of 1,032 records from stratified defined date boundaries in a
>> set. The latter is of interest because raw data can substantially change
>> structure or even format at points in time, and we need to be able to
>> sample within those data boundaries.
>>
>> Can anyone offer a link to an example of how nifi may be used to draw
>> samples randomly and/or in a systematic way from raw data collections?
>>
>
