I've seen a similar use case with DoubleClick/Google Analytics ( https://support.google.com/ds/answer/2791195?hl=en), where the reporting metrics have a "lookback window" of up to 30 days to mark conversion attribution (so if a user converts on the 14th day of clicking on an ad it will still be counted.
What we ended up doing in that case is setting the start/stop params of the query to the the full window (pulled daily) and then upserted in Redshift based on a primary key (in our case actually a composite key with multiple attributes). So you end up pulling a lot of redundant data but since there's no way to pull only updated records (which sounds like the case you're in), it's the best way to ensure your reporting is up-to-date. On Wed, Jun 6, 2018 at 1:10 PM Pedro Machado <pe...@205datalab.com> wrote: > Yes. It's literally the same API calls with the same dates, only done a few > days later. It's just redoing the same data pull but instead of pulling one > date each dag run, it would pull all dates for the previous week on > Tuesdays. > > Thanks! > -- [image: Astronomer Logo] <https://www.astronomer.io/> *Ben Gregory* Data Engineer Mobile: +1-615-483-3653 • Online: astronomer.io <https://www.astronomer.io/> Download our new ebook. <http://marketing.astronomer.io/guide/> From Volume to Value - A Guide to Data Engineering.
