Im importing data The data has a unique value, call it MD5 for now that could be a unique value for the data.
Each record that gets imported is converted to MD5, a lookup is done on the table for that MD5, if found it must leave it alone, if not found it must insert a new record... All the items in the table that was not imported must be deleted... The only feasible approach I have is to add a column to the table, like UPDATE_FLAG for example, During the import update_flag gets set to 0, Once a record is found update_flag gets set to 1, At the end of the import all records with update_flag = 0 gets deleted... However I did not want to add a column to the table, REASON being, I'm also version controlling the DB, and when an import occurs and nothing Has been added or removed, I don't want modifications to the DB, as the import can run many times over. I was considering the following: Create a temp table call it, Temp_ids for example Then insert into the temp table for each record that was found... At the end do something like Delete from imports where import_id not in (select id from temp_ids) But that might be horribly slow and memory expensive remembering that the import table may have millions of records.. What could the people here suggest to me, Thanks.
