Dear Team,
I am performing a batch operation. As per the use case, I cannot commit the
transaction from output operator to external system unless I process all
the batch data. Once all the data is processed, output operator will get a
bulking collection Object as single tuple. Output Operator is now supposed
to iterate over the collection and commit records to external system one by
one.
If the collection size is about 1K, then everything works fine, however, if
the size increases it causes the output operator to die. I am expecting
about 100K records in that collection which is served as a single tuple to
the output operator. I understand writing so many records in a single
execution of process method is causing issue. Can someone please suggest me
some approach to resolve this issue?
public final transient DefaultInputPort<T> input = new
DefaultInputPort<T>() {
public void process(List<String> tuple) { // write each object to
external system one by one }
};
