vincbeck commented on code in PR #74293:
URL: https://github.com/apache/airflow/pull/74293#discussion_r4338644987
##########
providers/amazon/src/airflow/providers/amazon/aws/triggers/s3.py:
##########
@@ -266,3 +270,141 @@ async def run(self) -> AsyncIterator[TriggerEvent]:
await asyncio.sleep(self.polling_period_seconds)
except Exception as e:
yield TriggerEvent({"status": "error", "message": str(e)})
+
+
+class S3KeyUpdateTrigger(BaseEventTrigger):
+ """
+ Fire an event whenever a single S3 object is updated.
+
+ Polls ``head_object`` for ``bucket_key`` and emits an event when the
object's ``ETag``
+ differs from the one the trigger last reported, which makes an upload to
that key usable
+ as a scheduling signal::
+
+ from airflow.sdk import Asset, AssetWatcher
+
+ report = Asset(
+ "daily_report",
+ watchers=[
+ AssetWatcher(
+ name="daily_report_updates",
+ trigger=S3KeyUpdateTrigger(bucket_name="my-bucket",
bucket_key="reports/daily.csv"),
+ )
+ ],
+ )
+
+
+ @dag(schedule=[report])
+ def downstream(): ...
+
+ The last-reported ``ETag`` is persisted in the asset state store, so a
triggerer restart
+ does not re-emit an unchanged object. On the first poll — and after a
restart when no
+ watermark was kept — the current object is itself the first event, so
``previous_etag`` is
+ ``None``. A task that must not run twice for one upload keys on ``etag``.
+
+ The event carries ``bucket_name``, ``bucket_key``, ``etag``,
``previous_etag``,
+ ``last_modified`` (ISO-8601) and ``size``. A missing key is not an error:
the trigger stays
+ silent and keeps polling until the object appears.
+
+ Updates are detected by ``ETag`` (the object's content fingerprint), so
re-uploading
+ byte-identical content does not fire. For a versioned bucket where every
upload must fire,
+ enable bucket versioning and the ``ETag`` still changes per upload.
Review Comment:
Good catch! I fixed the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
