On Sun, Sep 16, 2018 at 9:09 PM Olivier Flury via 4D_Tech < 4d_tech@lists.4d.com> wrote:
> Not really an answer to your question but a more general observation: > > Very often, when I have to do an export function towards other systems, I > end up in doing a bulk export. > > Either because I am lazy or the other side is lazy or both sides are lazy. > It's easy, just send everything. But sometimes it is worth, doing more > homework, on both sides, and export only what has changed since the last > export. Sometimes this is simply not possible (too complex, the other side > wants all data etc.), but very often it would be. And it would - most of > the time - result in a drastic reduction of amount of data to exchange. > Well observed. In this case, I have the good fortune to be working with source tables that include modification counters. (Think "sequence numbers" on each change.) This makes it *incredibly* easy to find all records that have been modified or added since the last push. For deletion, records in 4D are marked with a deleted date. I've always found this practice annoying and weird...and boy am I glad it's being used in this. With the delete date field in place (soft deleted records), it's simple to handle delete reconciliation. I don't push the delete date field, just a Boolean based on it. So, you push everything up including the delete flag, then a simple DELETE FROM table WHERE marked_for_deletion = true takes care of it. Amazingly easy. Welsh Harris and Tim Nevels get all the credit for setting things up this way in advance. Even still, there are sometimes still millions of records to move, particularly when we're first pushing a table over. In most cases, pushes are quite manageable in size on a day-to-day basis. These tables are getting pushed largely for aggregation and analysis, so it's not one-off operations. Push and keep pushing as more data flows through 4D. ********************************************************************** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **********************************************************************
