dstandish commented on issue #49900: URL: https://github.com/apache/airflow/issues/49900#issuecomment-2835888029
> The trigger retrieves a list of all objects in the object store with a “last updated” timestamp >= timestamp of the “last Asset Event”. Just a note. S3 has a `start_after_key` feature so it is optimal, for incremental processes from s3, if the keys are lexicographically sorted in order of arrival -- this way we do not need to list the whole directory and filter by timestamp in python (which can be slow and costly). Unless s3 changed, it does not have an analogous timestamp filter capability. Anyway, so tthen each run, you pass "last key" to `start_after_key` in s3 api and you get only the new fiiles. Other blob storages may or may not have similar feature. So we should not necessarily get tunnel vision on timestamps -- because they are not always optimal. Also I personally like to avoid using the word idempotent in Airflow discussions because it's not always clear exactly what we mean by that. It can mean different things in different contexts and I think clearer, more precise language is usually available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
