I have multiple ETL processes that are inserting data at the same time. Some
of the tables that this data is being inserted into have FK constraints that
must be maintained during the ETL process. See pseudocode below for an example:
create table employee (
employee_id integer,
bobm_dashlogistics schrieb am 07.06.2012 um 16:00 (-):
During the ETL process for Sales, we need to verify that the Employee
record already exists, and if not then we insert it before writing the
Sales record so that we can satisfy the FK. The same process happens
during the import of
Michael -
-Original Message-
From: firebird-support@yahoogroups.com
[mailto:firebird-support@yahoogroups.com] On Behalf Of Michael Ludwig
Sent: Thursday, June 07, 2012 1:12 PM
To: firebird-support@yahoogroups.com
Subject: Re: [firebird-support] Dealing with inserts from multiple
I think I would challenge whether adding a necessary entity (to satisfy
the FK requirement) needs to be in the same transaction as the overall
dataset processing.
If an Employee, for instance, can be added by any of several ETL
processes, does it really matter if one is added outside the main
Doug Chamberlin schrieb am 07.06.2012 um 14:07 (-0400):
I think I would challenge whether adding a necessary entity (to
satisfy the FK requirement) needs to be in the same transaction
as the overall dataset processing.
Another possibility might be to screen the dataset (which apparently
has to
On 6/7/12 3:57 PM, Michael Ludwig wrote:
Doug Chamberlin schrieb am 07.06.2012 um 14:07 (-0400):
I think I would challenge whether adding a necessary entity (to
satisfy the FK requirement) needs to be in the same transaction
as the overall dataset processing.
Another possibility might be to
Doug -
On June 07, 2012 6:08 PM Doug Chamberlin wrote:
Yes. On further thought I think I would make the whole ETL process a
two-pass affair. First pass check for existence of all needed
entities
and creates ones that are missing. Second pass should do the
inserting
proper without errors.
