These replies have all been very helpful and I wanted to get back to you and say thanks. We will have both situations to contend with: large numbers of flowfiles, each representing an atomic record or object, and smaller numbers of very large flowfiles from which we will draw a sample of records. I see suggestions in your answers that are going to help me get started. I don't yet have specific requirements for my sampling distribution and so was unable to provide that information. Though I am not certain I have all the requirements yet, I expect some of these large sets that have changed over time will have date boundaries marking where the data records change in some way. I intend to break down flowfiles based on those characteristics, and route different sets of records to different sampling processors. This will be the only way I can draw a meaningful sample over all these date thresholds. Thank you each again for offering much to work with.
On Fri, May 20, 2022 at 2:33 AM Hendrik Ruijter < [email protected]> wrote: > Your use case to draw a random sample of N records as one method of course > depends on the distribution you use. The uniform distribution documented in > the expression language guide is e.g. > > ${random():mod(10):plus(1)} returns random number between 1 and 10 > inclusive. > > There are numerous algorithms to create normal (Gaussian) distributions > from uniform distributions, e.g. Box-Müller. You can create lots of other > interesting distributions too. > > > > A RouteOnAttribute processor with two legs, one with your random sample, > and one with the entire flow should work. For example, > ${vodafone.items:equals(1)} would pick one flowfile from a uniform > distribution ${random():mod(1000):plus(1)} where the attribute is tested to > provide a 1 per mille sample. > > > > I would use an UpdatAttribute processor with state to sample a fixed > number of flowfiles but your use case is not detailed enough to answer at > present in my humble opinion. A typical pattern is to increment an index > > ${getStateValue("vodafone.items"):plus(${vodafone.pagesize})} in the > UpdateAttribute processor, next the RouteOnAttribute processor with two > legs: > > loop reset ${vodafone.items:equals(0)} > > , and loop.next > ${vodafone.items:gt(0):and(${vodafone.items:lt(${vodafone.total})})} > > > > *From:* James McMahon <[email protected]> > *Sent:* den 19 maj 2022 12:21 > *To:* [email protected] > *Subject:* NiFi to draw samples from very large raw data sets > > > > This e-mail was sent to you by someone outside the organization. Please > make sure it is a trusted contact before clicking on the links or > downloading any file. Protecting our systems is in your hands! > ------------------------------ > > I have been tasked to draw samples from very large raw data sets for > triage analysis. I am to provide multiple sampling methods. Drawing a > random sample of N records is one method. A second method is to draw a > fixed sample of 1,032 records from stratified defined date boundaries in a > set. The latter is of interest because raw data can substantially change > structure or even format at points in time, and we need to be able to > sample within those data boundaries. > > > > Can anyone offer a link to an example of how nifi may be used to draw > samples randomly and/or in a systematic way from raw data collections? >
