Owen-CH-Leung commented on code in PR #69897:
URL: https://github.com/apache/airflow/pull/69897#discussion_r4613967823


##########
providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py:
##########
@@ -973,7 +1033,14 @@ def upload(self, path: os.PathLike | str, ti: RuntimeTI | 
None = None) -> None:
         if not local_loc.is_file():
             return
 
-        log_id = _render_log_id(self.log_id_template, ti, ti.try_number)  # 
type: ignore[arg-type]
+        log_id_template = getattr(ti_context, "log_id_template", None) or 
self.log_id_template

Review Comment:
   The API sends `LogTemplate.elasticsearch_id`, which 
[`synchronize_log_template()`](https://github.com/apache/airflow/blob/13ffa73af98c2ed501b19d422c731d2dc50c35d1/airflow-core/src/airflow/utils/db.py#L1024)
 reads from `[elasticsearch] log_id_template`.
   When the two sections differ, this line replaces the configured OpenSearch 
template with the Elasticsearch template.
   Please save the OpenSearch template for each run. Add a test with different 
settings for the two providers.
   
   ---
   Drafted-by: Codex (GPT-6) (no human review before posting)



##########
providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py:
##########
@@ -1036,7 +1103,14 @@ def read(self, _relative_path: str, ti: RuntimeTI | 
None) -> tuple[LogSourceInfo
         if ti is None:
             # Logs are keyed by task instance, and ``upload`` skips logs 
without one.
             return [], []
-        log_id = _render_log_id(self.log_id_template, ti, ti.try_number)  # 
type: ignore[arg-type]
+        log_id_template, dag_run = _resolve_log_id_template(ti, 
self.log_id_template)

Review Comment:
   If a run pins template A and an older worker writes with configured template 
B, this reader queries A only.
   For legacy documents without `ti_id`, the alternate query also misses these 
logs. The previous reader queried B.
   Please try the configured template when the first query finds no logs. Add a 
regression test with legacy documents.
   
   ---
   Drafted-by: Codex (GPT-6) (no human review before posting)



##########
task-sdk/src/airflow/sdk/execution_time/supervisor.py:
##########
@@ -1942,7 +1946,7 @@ def _upload_logs(self):
 
         try:
             with _remote_logging_conn(self.client):
-                upload_to_remote(self.process_log, self.ti)
+                upload_to_remote(self.process_log, self.ti, 
ti_context=self._ti_context)

Review Comment:
   The upload in 
[`_fail_task_before_runtime()`](https://github.com/apache/airflow/blob/13ffa73af98c2ed501b19d422c731d2dc50c35d1/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L3134)
 still omits the `ti_context` returned by `task_instances.start()`.
   When a run pins A and the configuration changes to B, a launch failure 
writes its logs with B.
   Please pass `ti_context=ti_context` there too. Add a test with different 
saved and configured templates.
   
   ---
   Drafted-by: Codex (GPT-6) (no human review before posting)



##########
airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py:
##########
@@ -2071,6 +2074,76 @@ def 
test_ti_run_team_name_is_not_served_from_a_stale_cache(
         assert response.status_code == 200
         assert response.json()["dag_run"]["team_name"] == 
f"team-new-{dag_id[:8]}"
 
+    def test_ti_run_populates_log_id_template(
+        self, client, session, create_task_instance, create_log_template, 
time_machine
+    ):
+        """``log_id_template`` echoes the LogTemplate row pinned to the Dag 
run."""
+        instant = timezone.parse("2024-09-30T12:00:00Z")
+        time_machine.move_to(instant, tick=False)
+
+        custom_elasticsearch_id = 
"{dag_id}-{task_id}-{logical_date}-{try_number}"
+        create_log_template("{{ ti.dag_id }}.log", custom_elasticsearch_id)
+
+        # The Dag run pins the newest LogTemplate row at creation time.
+        ti = create_task_instance(
+            task_id="test_ti_run_populates_log_id_template",
+            state=State.QUEUED,
+            dagrun_state=DagRunState.RUNNING,
+            session=session,
+            start_date=instant,
+            dag_id=str(uuid4()),
+        )
+        session.commit()
+
+        response = client.patch(
+            f"/execution/task-instances/{ti.id}/run",
+            json={
+                "state": "running",
+                "hostname": "random-hostname",
+                "unixname": "random-unixname",
+                "pid": 100,
+                "start_date": instant.isoformat(),
+            },
+        )
+
+        assert response.status_code == 200
+        assert response.json()["log_id_template"] == custom_elasticsearch_id
+
+    def test_ti_run_populates_latest_log_id_template(
+        self, client, session, create_task_instance, create_log_template, 
time_machine
+    ):
+        """Each Dag run pins whichever LogTemplate row is newest when it is 
created."""
+        instant = timezone.parse("2024-09-30T12:00:00Z")
+        time_machine.move_to(instant, tick=False)
+
+        for i in range(3):
+            elasticsearch_id = 
f"{{dag_id}}-{{task_id}}-{{logical_date}}-{{try_number}}-v{i}"
+            create_log_template(f"{{{{ ti.dag_id }}}}-v{i}.log", 
elasticsearch_id)

Review Comment:
   This test starts each run before it creates the next template, so an 
implementation that always selects the latest template also passes.
   Create a run under A. Insert template B before you start that run.
   Check that the old run receives A. Check that a new run receives B.
   
   ---
   Drafted-by: Codex (GPT-6) (no human review before posting)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to