I think I would challenge whether adding a necessary entity (to satisfy
the FK requirement) needs to be in the same transaction as the overall
dataset processing.

If an Employee, for instance, can be added by any of several ETL
processes, does it really matter if one is added outside the main
transaction of the ETL dataset? If not, then make the adding of a
required entity happen in a separate transaction and commit that
transaction before continuing with the main ETL process. That way the
ETL process can continue and other concurrent ETL processes can also see
that new entity when they get to the point of needing it.

If one of the ETL processes ended up failing at some point perhaps a few
new entities were created and not removed during the rollback of the ETL
process but is that really such a problem?

Reply via email to