Hello,
The session is like a transaction, so you can call session.transfer as
many times as you want, but nothing is actually transferred until you
call session.commit.
If you are extending from AbstractProcessor it handles this for you like this...
try {
onTrigger(context, session);
session.commitAsync();
} catch (final Throwable t) {
session.rollback(true);
throw t;
}
The flow files in memory are not the contents of the flow files, they
are just pointers to where the content lives in the content
repository. The general process should be... read an input stream in
chunks/records, for each one write to an output stream, and repeat
until done, so that the whole input is never in memory and the whole
output is never in memory.
Thanks,
Bryan
On Wed, Jun 26, 2024 at 7:44 AM Jack Hinton
<jack.hin...@smartdcsit.co.uk> wrote:
>
> Hi Matt,
>
> Thanks for the reply! The use of rollback has solved that first issue
> handily. I'm having difficulties with the second one, however - it looks like
> all the other 'split' processors keep the resultant flowfiles in RAM until
> processing is complete in order to set the attribute I mentioned before, but
> this would seem to result in a processor that can potentially throttle all
> other operations by requiring more RAM than exists on the system (assuming
> the original flowfile is extremely large). Is there something they do that
> temporarily keeps the 'split' flowfiles in ROM whilst the processor is
> running that I'm missing? If so, how can I do the same?
>
> Thanks,
>
> Jack
