hansva commented on code in PR #9340:
URL: https://github.com/apache/hop/pull/9340#discussion_r4536781301


##########
plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageAsyncDispatcher.java:
##########
@@ -164,46 +291,174 @@ long getDroppedCount() {
     return droppedCount.get();
   }
 
-  /** Waits for the queue to drain (best effort). */
+  long getSentCount() {
+    return sentCount.get();
+  }
+
+  /** Delivery attempts that failed and will be (or were) tried again. */
+  long getFailedCount() {
+    return failedCount.get();
+  }
+
+  /** Events the collector refused; they are not resent. */
+  long getRejectedCount() {
+    return rejectedCount.get();
+  }
+
+  /** Events waiting in the disk-backed backlog (queue overflow plus earlier 
runs' spool files). */
+  int getBacklogSize() {
+    return backlog.size();
+  }
+
+  /**
+   * Waits for the queue and the disk-backed backlog to drain (best effort). 
The backlog is included
+   * so a short-lived process — a {@code hop-run} that replays what the 
previous one left — finishes
+   * its replay instead of handing every file on to the run after it.
+   */
   void awaitDrain(long timeoutMs) throws InterruptedException {
     long deadline = System.currentTimeMillis() + timeoutMs;
-    while (!queue.isEmpty() && System.currentTimeMillis() < deadline) {
+    while (System.currentTimeMillis() < deadline

Review Comment:
   `awaitDrain` only checks that the queue and backlog are empty, but the 
worker now takes events out of them (`drainTo(…, 49)` / `pollFirst`) before 
delivering. So `awaitDrain` can return while up to 50 events are still in 
flight, and `close()` then only gives `CLOSE_GRACE_MS`. With a slow collector 
and no spool those events are lost, and the 'still queued … are lost' count at 
line 428 doesn't include them, so the loss isn't reported either.



##########
plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageAsyncDispatcher.java:
##########
@@ -164,46 +291,174 @@ long getDroppedCount() {
     return droppedCount.get();
   }
 
-  /** Waits for the queue to drain (best effort). */
+  long getSentCount() {
+    return sentCount.get();
+  }
+
+  /** Delivery attempts that failed and will be (or were) tried again. */
+  long getFailedCount() {
+    return failedCount.get();
+  }
+
+  /** Events the collector refused; they are not resent. */
+  long getRejectedCount() {
+    return rejectedCount.get();
+  }
+
+  /** Events waiting in the disk-backed backlog (queue overflow plus earlier 
runs' spool files). */
+  int getBacklogSize() {
+    return backlog.size();
+  }
+
+  /**
+   * Waits for the queue and the disk-backed backlog to drain (best effort). 
The backlog is included
+   * so a short-lived process — a {@code hop-run} that replays what the 
previous one left — finishes
+   * its replay instead of handing every file on to the run after it.
+   */
   void awaitDrain(long timeoutMs) throws InterruptedException {
     long deadline = System.currentTimeMillis() + timeoutMs;
-    while (!queue.isEmpty() && System.currentTimeMillis() < deadline) {
+    while (System.currentTimeMillis() < deadline
+        && (!queue.isEmpty() || (!backlog.isEmpty() && backlogRetryAtMs < 
deadline))) {
+      // A backlog that is backing off past the deadline will not be sent in 
time; it stays on
+      // disk for the next start, so there is no point in waiting for it.
       Thread.sleep(20);
     }
   }
 
   private void runWorker() {
-    while (running.get() || !queue.isEmpty()) {
+    while ((running.get() || !queue.isEmpty()) && !stopped.get()) {
       try {
-        RunEvent event = queue.poll(250, TimeUnit.MILLISECONDS);
-        if (event != null) {
-          if (emitter.emit(event)) {
-            sentCount.incrementAndGet();
-          } else {
-            failedCount.incrementAndGet();
+        // With backlog work waiting, check the queue without blocking: 
otherwise every backlog
+        // event would wait out the poll timeout and a replay would crawl at a 
few events a second.
+        QueuedEvent first = backlogReady() ? queue.poll() : queue.poll(250, 
TimeUnit.MILLISECONDS);
+        List<QueuedEvent> batch = new ArrayList<>();
+        boolean fromBacklog = first == null;
+        if (first != null) {
+          batch.add(first);
+          queue.drainTo(batch, MAX_EMIT_BATCH - 1);
+        } else {
+          QueuedEvent next;
+          while (batch.size() < MAX_EMIT_BATCH && (next = pollBacklog()) != 
null) {
+            batch.add(next);
           }
         }
+        if (!batch.isEmpty()) {
+          deliver(batch, fromBacklog);
+        }
       } catch (InterruptedException e) {
         Thread.currentThread().interrupt();
         break;
       }
     }
   }
 
+  /**
+   * The next spooled event to retry, or {@code null} while the queue is the 
thing to serve, the
+   * backlog is empty, or a failed backlog delivery is still backing off.
+   */
+  private QueuedEvent pollBacklog() {
+    return backlogReady() ? backlog.pollFirst() : null;
+  }
+
+  private boolean backlogReady() {
+    return running.get() && !backlog.isEmpty() && System.currentTimeMillis() 
>= backlogRetryAtMs;
+  }
+
+  private void deliver(List<QueuedEvent> batch, boolean fromBacklog) {
+    List<OpenLineageEmitter.Result> results;
+    try {
+      results = 
emitter.emitAll(batch.stream().map(QueuedEvent::event).toList());
+    } catch (RuntimeException e) {
+      log.logError("Lineage event delivery failed unexpectedly: " + 
e.getMessage(), e);
+      results = null;
+    }
+    for (int i = 0; i < batch.size(); i++) {
+      OpenLineageEmitter.Result result =
+          results != null && results.size() == batch.size()
+              ? results.get(i)
+              : OpenLineageEmitter.Result.RETRY;
+      settle(batch.get(i), result, fromBacklog);
+    }
+  }
+
+  private void settle(QueuedEvent queued, OpenLineageEmitter.Result result, 
boolean fromBacklog) {
+    if (result == OpenLineageEmitter.Result.SENT) {
+      sentCount.incrementAndGet();
+      if (queued.spoolId() != null) {
+        spool.delete(queued.spoolId());
+      }
+      backlogRetryAtMs = 0;
+    } else if (result == OpenLineageEmitter.Result.REJECTED) {
+      // Refused for what it is: resending can only fail the same way, and 
keeping it would make
+      // every replay start with it.
+      rejectedCount.incrementAndGet();
+      if (queued.spoolId() != null) {
+        spool.delete(queued.spoolId());
+      }
+    } else {
+      failedCount.incrementAndGet();
+      if (fromBacklog) {
+        // Still on disk, so nothing is lost. It goes to the back of the 
backlog: an event that
+        // keeps failing must not stop the ones behind it, and the pause keeps 
a dead collector
+        // from being hammered with every file in turn.
+        backlog.addLast(queued);

Review Comment:
   A failed delivery is re-added with `backlog.addLast`, behind events of the 
same run that were queued later. And line 250 only routes new events to the 
backlog while it's non-empty, but the in-flight batch has already been polled 
out of it, so a new event can skip ahead through the queue. Example: START of 
run R fails with a 503 and goes to the back; R's COMPLETE is delivered first. 
Marquez/DataHub then leave the run RUNNING or record the wrong end time. This 
contradicts the ordering promise in the comment at line 251.



##########
plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageAsyncDispatcher.java:
##########
@@ -164,46 +291,174 @@ long getDroppedCount() {
     return droppedCount.get();
   }
 
-  /** Waits for the queue to drain (best effort). */
+  long getSentCount() {
+    return sentCount.get();
+  }
+
+  /** Delivery attempts that failed and will be (or were) tried again. */
+  long getFailedCount() {
+    return failedCount.get();
+  }
+
+  /** Events the collector refused; they are not resent. */
+  long getRejectedCount() {
+    return rejectedCount.get();
+  }
+
+  /** Events waiting in the disk-backed backlog (queue overflow plus earlier 
runs' spool files). */
+  int getBacklogSize() {
+    return backlog.size();
+  }
+
+  /**
+   * Waits for the queue and the disk-backed backlog to drain (best effort). 
The backlog is included
+   * so a short-lived process — a {@code hop-run} that replays what the 
previous one left — finishes
+   * its replay instead of handing every file on to the run after it.
+   */
   void awaitDrain(long timeoutMs) throws InterruptedException {
     long deadline = System.currentTimeMillis() + timeoutMs;
-    while (!queue.isEmpty() && System.currentTimeMillis() < deadline) {
+    while (System.currentTimeMillis() < deadline
+        && (!queue.isEmpty() || (!backlog.isEmpty() && backlogRetryAtMs < 
deadline))) {
+      // A backlog that is backing off past the deadline will not be sent in 
time; it stays on
+      // disk for the next start, so there is no point in waiting for it.
       Thread.sleep(20);
     }
   }
 
   private void runWorker() {
-    while (running.get() || !queue.isEmpty()) {
+    while ((running.get() || !queue.isEmpty()) && !stopped.get()) {
       try {
-        RunEvent event = queue.poll(250, TimeUnit.MILLISECONDS);
-        if (event != null) {
-          if (emitter.emit(event)) {
-            sentCount.incrementAndGet();
-          } else {
-            failedCount.incrementAndGet();
+        // With backlog work waiting, check the queue without blocking: 
otherwise every backlog
+        // event would wait out the poll timeout and a replay would crawl at a 
few events a second.
+        QueuedEvent first = backlogReady() ? queue.poll() : queue.poll(250, 
TimeUnit.MILLISECONDS);
+        List<QueuedEvent> batch = new ArrayList<>();
+        boolean fromBacklog = first == null;
+        if (first != null) {
+          batch.add(first);
+          queue.drainTo(batch, MAX_EMIT_BATCH - 1);
+        } else {
+          QueuedEvent next;
+          while (batch.size() < MAX_EMIT_BATCH && (next = pollBacklog()) != 
null) {
+            batch.add(next);
           }
         }
+        if (!batch.isEmpty()) {
+          deliver(batch, fromBacklog);
+        }
       } catch (InterruptedException e) {
         Thread.currentThread().interrupt();
         break;
       }
     }
   }
 
+  /**
+   * The next spooled event to retry, or {@code null} while the queue is the 
thing to serve, the
+   * backlog is empty, or a failed backlog delivery is still backing off.
+   */
+  private QueuedEvent pollBacklog() {
+    return backlogReady() ? backlog.pollFirst() : null;
+  }
+
+  private boolean backlogReady() {
+    return running.get() && !backlog.isEmpty() && System.currentTimeMillis() 
>= backlogRetryAtMs;
+  }
+
+  private void deliver(List<QueuedEvent> batch, boolean fromBacklog) {
+    List<OpenLineageEmitter.Result> results;
+    try {
+      results = 
emitter.emitAll(batch.stream().map(QueuedEvent::event).toList());

Review Comment:
   A batch of up to 50 events goes to `emitAll`, whose default 
(`OpenLineageEmitter.java:72`) sends each event separately with the full HTTP 
retry/backoff cycle, and nothing stops the batch after the first RETRY. With 
the collector down and default settings (5s connect timeout, 3 retries, 
0.5/1/2s backoff) that's ~23s per event, so ~20 minutes per batch of 50. 
Meanwhile the single worker serves nothing else: with DROP and no spool the 10k 
queue fills and live lineage is dropped, and the log gets 50 'Failed to POST' 
errors per batch. Could the batch bail out (and back off) on the first 
retryable failure?



##########
plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageAsyncDispatcher.java:
##########
@@ -164,46 +291,174 @@ long getDroppedCount() {
     return droppedCount.get();
   }
 
-  /** Waits for the queue to drain (best effort). */
+  long getSentCount() {
+    return sentCount.get();
+  }
+
+  /** Delivery attempts that failed and will be (or were) tried again. */
+  long getFailedCount() {
+    return failedCount.get();
+  }
+
+  /** Events the collector refused; they are not resent. */
+  long getRejectedCount() {
+    return rejectedCount.get();
+  }
+
+  /** Events waiting in the disk-backed backlog (queue overflow plus earlier 
runs' spool files). */
+  int getBacklogSize() {
+    return backlog.size();
+  }
+
+  /**
+   * Waits for the queue and the disk-backed backlog to drain (best effort). 
The backlog is included
+   * so a short-lived process — a {@code hop-run} that replays what the 
previous one left — finishes
+   * its replay instead of handing every file on to the run after it.
+   */
   void awaitDrain(long timeoutMs) throws InterruptedException {
     long deadline = System.currentTimeMillis() + timeoutMs;
-    while (!queue.isEmpty() && System.currentTimeMillis() < deadline) {
+    while (System.currentTimeMillis() < deadline
+        && (!queue.isEmpty() || (!backlog.isEmpty() && backlogRetryAtMs < 
deadline))) {
+      // A backlog that is backing off past the deadline will not be sent in 
time; it stays on
+      // disk for the next start, so there is no point in waiting for it.
       Thread.sleep(20);
     }
   }
 
   private void runWorker() {
-    while (running.get() || !queue.isEmpty()) {
+    while ((running.get() || !queue.isEmpty()) && !stopped.get()) {
       try {
-        RunEvent event = queue.poll(250, TimeUnit.MILLISECONDS);
-        if (event != null) {
-          if (emitter.emit(event)) {
-            sentCount.incrementAndGet();
-          } else {
-            failedCount.incrementAndGet();
+        // With backlog work waiting, check the queue without blocking: 
otherwise every backlog
+        // event would wait out the poll timeout and a replay would crawl at a 
few events a second.
+        QueuedEvent first = backlogReady() ? queue.poll() : queue.poll(250, 
TimeUnit.MILLISECONDS);
+        List<QueuedEvent> batch = new ArrayList<>();
+        boolean fromBacklog = first == null;
+        if (first != null) {
+          batch.add(first);
+          queue.drainTo(batch, MAX_EMIT_BATCH - 1);
+        } else {
+          QueuedEvent next;
+          while (batch.size() < MAX_EMIT_BATCH && (next = pollBacklog()) != 
null) {
+            batch.add(next);
           }
         }
+        if (!batch.isEmpty()) {
+          deliver(batch, fromBacklog);
+        }
       } catch (InterruptedException e) {
         Thread.currentThread().interrupt();
         break;
       }
     }
   }
 
+  /**
+   * The next spooled event to retry, or {@code null} while the queue is the 
thing to serve, the
+   * backlog is empty, or a failed backlog delivery is still backing off.
+   */
+  private QueuedEvent pollBacklog() {
+    return backlogReady() ? backlog.pollFirst() : null;
+  }
+
+  private boolean backlogReady() {
+    return running.get() && !backlog.isEmpty() && System.currentTimeMillis() 
>= backlogRetryAtMs;
+  }
+
+  private void deliver(List<QueuedEvent> batch, boolean fromBacklog) {
+    List<OpenLineageEmitter.Result> results;
+    try {
+      results = 
emitter.emitAll(batch.stream().map(QueuedEvent::event).toList());
+    } catch (RuntimeException e) {
+      log.logError("Lineage event delivery failed unexpectedly: " + 
e.getMessage(), e);
+      results = null;
+    }
+    for (int i = 0; i < batch.size(); i++) {
+      OpenLineageEmitter.Result result =
+          results != null && results.size() == batch.size()
+              ? results.get(i)
+              : OpenLineageEmitter.Result.RETRY;
+      settle(batch.get(i), result, fromBacklog);
+    }
+  }
+
+  private void settle(QueuedEvent queued, OpenLineageEmitter.Result result, 
boolean fromBacklog) {
+    if (result == OpenLineageEmitter.Result.SENT) {
+      sentCount.incrementAndGet();
+      if (queued.spoolId() != null) {
+        spool.delete(queued.spoolId());
+      }
+      backlogRetryAtMs = 0;
+    } else if (result == OpenLineageEmitter.Result.REJECTED) {
+      // Refused for what it is: resending can only fail the same way, and 
keeping it would make
+      // every replay start with it.
+      rejectedCount.incrementAndGet();
+      if (queued.spoolId() != null) {
+        spool.delete(queued.spoolId());
+      }
+    } else {
+      failedCount.incrementAndGet();
+      if (fromBacklog) {
+        // Still on disk, so nothing is lost. It goes to the back of the 
backlog: an event that
+        // keeps failing must not stop the ones behind it, and the pause keeps 
a dead collector
+        // from being hammered with every file in turn.
+        backlog.addLast(queued);
+        backlogRetryAtMs = System.currentTimeMillis() + backlogRetryPauseMs;
+      } else if (queued.spoolId() != null) {
+        // A live event that is on disk is retried by this process too, not 
only by the next one:
+        // a long-running server would otherwise keep it until a restart, and 
a full spool would
+        // quietly stop protecting new events.
+        backlog.addLast(queued);
+        backlogRetryAtMs = System.currentTimeMillis() + backlogRetryPauseMs;
+      }
+    }
+  }
+
   @Override
   public void close() {
     running.set(false);
     if (metricsScheduler != null) {
       metricsScheduler.shutdownNow();
     }
     try {
-      worker.join(30_000);
+      worker.join(CLOSE_GRACE_MS);
     } catch (InterruptedException e) {
       Thread.currentThread().interrupt();
     }
+    stopped.set(true);
+    long notSpooled = queue.stream().filter(queued -> queued.spoolId() == 
null).count();
+    if (notSpooled > 0) {
+      log.logError(
+          notSpooled
+              + " lineage event(s) were still queued when the sink shut down 
and are lost; set a"
+              + " spool folder to keep them for the next start");
+    }
     long dropped = droppedCount.get();
     if (dropped > 0) {
-      log.logError("OpenLineage sink dropped " + dropped + " events due to 
buffer overflow");
+      log.logError("Lineage sink dropped " + dropped + " events due to buffer 
overflow");
+    }
+    if (spool != null) {
+      long remaining = spool.getStoredCount();
+      if (remaining > 0) {
+        log.logError(
+            "Lineage sink shut down with "
+                + remaining
+                + " event(s) still spooled in "
+                + spool.getLocation()
+                + "; they will be sent on the next start");
+      }
+      if (spool.getFullRefusals() > 0 || spool.getWriteFailures() > 0) {
+        log.logError(
+            "Lineage spool metrics: stored="
+                + remaining
+                + " event(s)/"
+                + spool.getStoredBytes()
+                + " bytes, refused (spool full)="
+                + spool.getFullRefusals()
+                + ", unwritable="
+                + spool.getWriteFailures()
+                + ", unreadable on replay="
+                + spool.getUnreadableFiles());
+      }
     }
     emitter.close();

Review Comment:
   `emitter.close()` runs after a bounded `worker.join(CLOSE_GRACE_MS)` without 
checking that the worker actually stopped. If the worker is mid-retry against 
an unreachable collector (see the comment on line 370, that can last minutes), 
the HTTP client is closed while `emit()` is still using it on the worker 
thread, giving IllegalStateException / connection pool errors and a spool 
delete racing the teardown.



##########
plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageAsyncDispatcher.java:
##########
@@ -81,15 +128,73 @@ enum OverflowPolicy {
       ILogChannel log,
       long metricsIntervalMs,
       long enqueueTimeoutMs) {
+    this(bufferSize, overflowPolicy, emitter, log, metricsIntervalMs, 
enqueueTimeoutMs, null);
+  }
+
+  OpenLineageAsyncDispatcher(
+      int bufferSize,
+      OverflowPolicy overflowPolicy,
+      OpenLineageEmitter emitter,
+      ILogChannel log,
+      long metricsIntervalMs,
+      long enqueueTimeoutMs,
+      OpenLineageSpool spool) {
+    this(
+        bufferSize,
+        overflowPolicy,
+        emitter,
+        log,
+        metricsIntervalMs,
+        enqueueTimeoutMs,
+        spool,
+        DEFAULT_BACKLOG_RETRY_PAUSE_MS);
+  }
+
+  OpenLineageAsyncDispatcher(
+      int bufferSize,
+      OverflowPolicy overflowPolicy,
+      OpenLineageEmitter emitter,
+      ILogChannel log,
+      long metricsIntervalMs,
+      long enqueueTimeoutMs,
+      OpenLineageSpool spool,
+      long backlogRetryPauseMs) {
     this.queue = new LinkedBlockingQueue<>(bufferSize);
     this.overflowPolicy = overflowPolicy;
     this.enqueueTimeoutMs = enqueueTimeoutMs;
     this.emitter = emitter;
+    this.spool = spool;
+    this.backlogRetryPauseMs = backlogRetryPauseMs;
     this.log = log;
     this.worker = new Thread(this::runWorker, "hop-openlineage-sink");
     this.worker.setDaemon(true);
     this.worker.start();
     this.metricsScheduler = startMetricsScheduler(metricsIntervalMs);
+    replaySpoolFromAnEarlierRun();
+  }
+
+  /**
+   * Hands the events an earlier process left on disk to the backlog. They are 
already durable, so
+   * this only schedules their delivery: the worker keeps up with the 
in-memory queue first and
+   * picks the backlog up when that is empty.
+   */
+  private void replaySpoolFromAnEarlierRun() {
+    if (spool == null) {
+      return;
+    }
+    List<OpenLineageSpool.SpooledEvent> stored = spool.loadAll();

Review Comment:
   On init every process replays every file in the spool folder, including 
files another live process sharing the folder has stored but not delivered yet 
(the spool code already accounts for files that 'disappear'). E.g. hop-server 
and a hop-run sharing `HOP_LINEAGE_OPENLINEAGE_SPOOL_DIR`: hop-run sends the 
server's in-flight events too, so they're duplicated and possibly reordered, 
and hop-run's shutdown `awaitDrain` can wait up to 30s on another process's 
backlog. A per-process subfolder or a lock/claim on files would avoid this.



##########
plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageSinkConfig.java:
##########
@@ -45,9 +46,21 @@ public final class OpenLineageSinkConfig {
       "HOP_LINEAGE_OPENLINEAGE_TRUSTSTORE_PASSWORD";
   public static final String VAR_KEYSTORE_TYPE = 
"HOP_LINEAGE_OPENLINEAGE_KEYSTORE_TYPE";
   public static final String VAR_SQL_PARSE = 
"HOP_LINEAGE_OPENLINEAGE_SQL_PARSE";
-  public static final String VAR_FILE_SPEC_NAMING = 
"HOP_LINEAGE_OPENLINEAGE_FILE_SPEC_NAMING";
+  public static final String VAR_TRANSFORM_JOBS = 
"HOP_LINEAGE_OPENLINEAGE_TRANSFORM_JOBS";

Review Comment:
   `HOP_LINEAGE_OPENLINEAGE_FILE_SPEC_NAMING` is removed and `fileDatasetId` 
(`OpenLineageEventMapper.java:1210`) now always uses spec naming. The old 
variable's own description said spec naming changes existing file dataset 
identities, and it defaulted to off. So a deployment upgrading gets new 
namespace/name values for every file dataset in Marquez/DataHub, existing 
graphs split into old and new nodes, and a variable they may have set 
explicitly is silently ignored. Could we keep an opt-out, or at least log a 
warning when the old variable is set, and mention it in the upgrade notes?



##########
plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageSpool.java:
##########
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage.openlineage;
+
+import io.openlineage.client.OpenLineage.RunEvent;
+import io.openlineage.client.OpenLineageClientUtils;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Pattern;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileType;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.vfs.HopVfs;
+
+/**
+ * Local write-ahead spool for outbound OpenLineage events.
+ *
+ * <p>An event is written to its own file before it is handed to the 
dispatcher, and the file is
+ * deleted only once the collector accepted it (HTTP 2xx). Whatever is still 
on disk when the sink
+ * starts is replayed first, so a collector restart, a network outage or a 
process that dies between
+ * "pipeline finished" and "HTTP 2xx" delays lineage instead of losing it.
+ *
+ * <p>Delivery is therefore <i>at least once</i>: a process that dies after a 
successful POST but
+ * before the delete replays that one event. OpenLineage events set aspects on 
entities rather than
+ * appending records, so a replay rewrites the same state; a duplicate is the 
cheaper problem
+ * compared to a permanent gap in the graph.
+ *
+ * <p>The spool is bounded by {@code maxBytes}; when it is full new events are 
<i>not</i> spooled
+ * (they are still sent) and the refusal is counted, so a collector outage 
cannot fill the disk. The
+ * oldest events are kept: they are the ones no later run will re-emit.
+ */
+final class OpenLineageSpool {
+
+  static final String FILE_SUFFIX = ".json";
+
+  /** Written first and renamed into place, so a crash never leaves a half 
event to replay. */
+  static final String TEMP_SUFFIX = ".tmp";
+
+  /**
+   * The names this spool writes ({@link #nextName()}). Only these are listed, 
replayed or deleted:
+   * a spool pointed at a folder that holds other files must never touch them.
+   */
+  private static final Pattern SPOOL_FILE_NAME =
+      Pattern.compile(
+          "\\d{13}-\\d{8}-"
+              + "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"
+              + Pattern.quote(FILE_SUFFIX));
+
+  /** A spooled event, identified by the name of the file that holds it. */
+  record SpooledEvent(String id, RunEvent event) {}
+
+  private final String location;
+  private final long maxBytes;
+  private final ILogChannel log;
+  private final AtomicLong storedBytes = new AtomicLong();
+  private final AtomicLong storedFiles = new AtomicLong();
+
+  /**
+   * Size of every file this instance counts, so a file that disappears 
(another process sharing the
+   * folder delivered it first) still leaves the totals right.
+   */
+  private final Map<String, Long> storedSizes = new ConcurrentHashMap<>();
+
+  private final AtomicLong sequence = new AtomicLong();
+  private final AtomicLong serialisationFailures = new AtomicLong();
+  private final AtomicLong writeFailures = new AtomicLong();
+  private final AtomicLong fullRefusals = new AtomicLong();
+  private final AtomicLong unreadableFiles = new AtomicLong();
+  private volatile boolean fullReported;
+
+  /**
+   * Opens a spool on {@code directory}, or returns {@code null} when no 
directory is configured or
+   * it cannot be used. A misconfigured spool must not stop lineage from being 
sent: the caller gets
+   * {@code null} and sends without durability, with the problem logged here.
+   */
+  static OpenLineageSpool open(String directory, long maxBytes, ILogChannel 
log) {
+    if (StringUtils.isBlank(directory)) {
+      return null;
+    }
+    try {
+      return new OpenLineageSpool(HopVfs.getFileObject(directory.trim()), 
maxBytes, log);
+    } catch (Exception e) {
+      log.logError(
+          "Lineage spool directory '"
+              + directory
+              + "' is not usable; lineage will be sent without a spool: "
+              + e.getMessage(),
+          e);
+      return null;
+    }
+  }
+
+  private OpenLineageSpool(FileObject folder, long maxBytes, ILogChannel log) 
throws Exception {
+    this.location = folder.getName().getURI();
+    this.maxBytes = maxBytes;
+    this.log = log;
+    if (!folder.exists()) {
+      folder.createFolder();
+    }
+    if (!folder.isFolder()) {
+      throw new IllegalArgumentException("not a folder: " + location);
+    }
+    for (String name : spoolFileNames()) {

Review Comment:
   The constructor lists the folder and stats every file, then 
`replaySpoolFromAnEarlierRun` calls `loadAll`, which lists it again and parses 
every file into RunEvents kept in an unbounded in-memory backlog, all 
synchronously during sink init on the hub thread. After a long outage (64MB 
default cap, ~100k small files) that's two full scans plus 100k parses before 
the hub dispatches anything, and memory several times the JSON size. Streaming 
the replay lazily from the file names would avoid both.



##########
plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageLineageSink.java:
##########
@@ -117,27 +157,40 @@ public void accept(List<LineageEvent> events) throws 
HopException {
   @Override
   public void shutdown() throws HopException {
     if (dispatcher != null) {
+      // A dataset observation whose run never reported its end has no run 
event left to carry it.
+      // Send those as standalone events before the queue drains, so that 
nothing observed is
+      // dropped.
+      List<RunEvent> carriedNowhere = mapper != null ? 
mapper.flushObservations() : List.of();

Review Comment:
   `shutdown()` runs `mapper.flushObservations()` and then nulls `mapper`, 
`config` and `dispatcher`, but `LineageHub.shutdown` only joins its worker for 
5s, the same as `DEFAULT_ENQUEUE_TIMEOUT_MS`, so `accept()` can still be 
running `mapBatch` on the hub worker. The mapper state 
(`PipelineRunState.transformIo`, `deferredTerminals`, `PendingRun` lists) isn't 
thread-safe and `accept()` has no null guard. With BLOCK and a full queue at 
exit you get a ConcurrentModificationException in `flushObservations`, or an 
NPE at line 151 when the worker resumes after `mapper = null`.



##########
plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageAsyncDispatcher.java:
##########
@@ -108,53 +213,75 @@ private ScheduledExecutorService 
startMetricsScheduler(long intervalMs) {
   }
 
   private void logMetrics() {
-    log.logBasic(
-        "OpenLineage sink metrics: sent="
-            + sentCount.get()
-            + ", failed="
-            + failedCount.get()
-            + ", dropped="
-            + droppedCount.get()
-            + ", queued="
-            + queue.size());
+    StringBuilder message =
+        new StringBuilder("Lineage delivery metrics: sent=")
+            .append(sentCount.get())
+            .append(", failed=")
+            .append(failedCount.get())
+            .append(", rejected=")
+            .append(rejectedCount.get())
+            .append(", dropped=")
+            .append(droppedCount.get())
+            .append(", queued=")
+            .append(queue.size())
+            .append(", spooled=")
+            .append(spool == null ? "off" : 
String.valueOf(spool.getStoredCount()));
+    if (spool != null) {
+      message.append(", backlog=").append(backlog.size());
+    }
+    log.logBasic(message.toString());
   }
 
   /**
-   * Enqueues events.
+   * Enqueues events, persisting them first when a spool is configured.
    *
    * <p>The caller is the lineage hub's single dispatcher thread, and the hub 
only regains that
    * thread — and with it the ability to process the flush marker a finishing 
pipeline is waiting on
    * — once this returns. An unbounded wait here therefore does not merely 
delay lineage: it stalls
    * every pipeline and workflow completion for the hub's full flush timeout. 
So {@link
    * OverflowPolicy#BLOCK} waits at most {@link #enqueueTimeoutMs} for 
capacity and then drops
-   * (counted), and {@link OverflowPolicy#DROP} drops at once.
+   * (counted), and {@link OverflowPolicy#DROP} drops at once. With a spool, 
"drops" are only the
+   * events that could not be written to disk either: an event that is on disk 
but did not fit the
+   * queue waits in the backlog instead.
    */
   void enqueueAll(List<RunEvent> events) {
     for (RunEvent event : events) {
-      try {
-        boolean queued =
-            overflowPolicy == OverflowPolicy.BLOCK
-                ? queue.offer(event, enqueueTimeoutMs, TimeUnit.MILLISECONDS)
-                : queue.offer(event);
-        if (!queued) {
+      QueuedEvent queued = new QueuedEvent(event, spool == null ? null : 
spool.store(event));

Review Comment:
   `enqueueAll` runs on the LineageHub's single dispatcher thread, and now 
serialises every event to JSON and does a synchronous VFS create/write/move per 
event when the spool is on. That's the same thread that has to process the 
flush markers pipelines wait on. With a Pipeline Executor running a 
sub-pipeline per row (or a spool on `s3://` / `sftp://`, which 
`HopVfs.getFileObject` allows), the hub thread falls behind, the hub's bounded 
queue starts dropping events and each pipeline completion's `flush()` waits up 
to its 60s timeout. Could the spool writes move to the dispatcher's own worker?



##########
plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageEventMapper.java:
##########
@@ -237,26 +572,540 @@ private Optional<RunEvent> mapFileIo(LineageEvent event) 
{
     if (!(payload instanceof FileIoLineagePayload fileIo)) {
       return Optional.empty();
     }
-    LineageContext ctx = event.getContext();
 
     List<InputDataset> inputs = new ArrayList<>();
     List<OutputDataset> outputs = new ArrayList<>();
     addFileIoDatasets(fileIo, inputs, outputs);
+    return observe(event, inputs, outputs, null);
+  }
+
+  /**
+   * Records the datasets an I/O observation found against the run it belongs 
to, so that run's
+   * terminal event can carry them. Returns an event only when there is no run 
left to carry them:
+   * an observation that arrives after its run reported COMPLETE or FAIL has 
nothing ahead of it,
+   * and is sent as a standalone dataset event.
+   */
+  private Optional<RunEvent> observe(
+      LineageEvent event, List<InputDataset> inputs, List<OutputDataset> 
outputs, String sqlText) {
     if (inputs.isEmpty() && outputs.isEmpty()) {
       return Optional.empty();
     }
+    LineageContext ctx = event.getContext();
+    boolean perTransform = isTransformOfAPipelineRun(ctx);
+    if (perTransform && 
!endedPipelineRuns.contains(pipelineLogChannelId(ctx))) {
+      // Copies of a transform share one job, so they share one record of what 
it touched.
+      pipelineRun(ctx)
+          .transformIo
+          .computeIfAbsent(
+              ctx.getTransformName(),
+              name ->
+                  new PendingRun(
+                      resolveJobName(ctx),
+                      resolveRunId(ctx, event.getEventId()),
+                      JobIdentity.of(ctx)))
+          .add(inputs, outputs, sqlText);
+      return Optional.empty();
+    }
+    String jobName = perTransform ? resolveJobName(ctx) : 
resolveFileIoJobName(ctx);
+    String runId =
+        perTransform
+            ? resolveRunId(ctx, event.getEventId())
+            : resolveFileIoRunId(ctx, event.getEventId());
+    JobIdentity identity = perTransform ? JobIdentity.of(ctx) : 
JobIdentity.ioOwner(ctx);
+    String key = runKey(jobName, runId);
+    if (perTransform || endedRuns.contains(key)) {
+      PendingRun late = new PendingRun(jobName, runId, identity);
+      late.add(inputs, outputs, sqlText);
+      return Optional.of(standaloneEvent(late, inputs, 
event.getTimestampMillis()));
+    }
+    pendingRuns
+        .computeIfAbsent(key, k -> new PendingRun(jobName, runId, identity))
+        .add(inputs, outputs, sqlText);
+    return Optional.empty();
+  }
 
-    ZonedDateTime eventTime = toUtc(event.getTimestampMillis());
-    RunEvent runEvent =
-        ol.newRunEventBuilder()
-            .eventType(EventType.OTHER)
-            .eventTime(eventTime)
-            .run(runNode(resolveFileIoRunId(ctx, event.getEventId()), 
eventTime, null))
-            
.job(ol.newJobBuilder().namespace(namespace).name(resolveFileIoJobName(ctx)).build())
-            .inputs(inputs.isEmpty() ? null : inputs)
-            .outputs(outputs.isEmpty() ? null : outputs)
-            .build();
-    return Optional.of(runEvent);
+  /**
+   * Whether this event belongs to a transform job of its own rather than to 
its pipeline's: only
+   * when the mapping is switched on and the transform's pipeline run is known.
+   */
+  private boolean isTransformOfAPipelineRun(LineageContext ctx) {
+    return transformJobs
+        && ctx != null
+        && ctx.getSubjectType() == LineageSubjectType.TRANSFORM
+        && !Utils.isEmpty(ctx.getTransformName())
+        && !Utils.isEmpty(pipelineLogChannelId(ctx));
+  }
+
+  private PipelineRunState pipelineRun(LineageContext transformCtx) {
+    return pipelineRun(
+        pipelineLogChannelId(transformCtx),
+        resolveFileIoJobName(transformCtx),
+        JobIdentity.ioOwner(transformCtx));
+  }
+
+  private PipelineRunState pipelineRun(String pipelineRunId, String jobName, 
JobIdentity identity) {
+    return pipelineRuns.computeIfAbsent(
+        pipelineRunId, id -> new PipelineRunState(id, jobName, identity));
+  }
+
+  /**
+   * Gives each held transform terminal event its datasets, now that the 
pipeline is over.
+   *
+   * <p>With the run's hop graph, a transform's job gets what it read plus 
what its upstream
+   * transforms produced, so each of its dataset pairs is a real one. Without 
a graph (an engine
+   * that predates the graph event) a transform job could hold only one end of 
a pair and no edge
+   * would be derived, so the datasets go to the pipeline's run instead, as 
with transform jobs off.
+   *
+   * @param pipelineRun the pipeline run's own observations; receives the 
datasets without a graph
+   * @return the transform events, to be sent before the pipeline's own 
terminal event
+   */
+  private List<RunEvent> finishPipelineRun(PipelineRunState state, PendingRun 
pipelineRun) {
+    List<RunEvent> events = new ArrayList<>();
+    boolean propagate = state.graph != null;
+    Set<String> finished = new LinkedHashSet<>();
+    for (LineageEvent terminal : state.deferredTerminals) {
+      LineageContext ctx = terminal.getContext();
+      String transform = ctx.getTransformName();
+      finished.add(transform);
+      endedRuns.add(runKey(resolveJobName(ctx), resolveRunId(ctx, 
terminal.getEventId())));
+      PendingRun own = state.transformIo.get(transform);
+      if (propagate) {
+        List<InputDataset> ownInputs = own == null ? List.of() : 
own.getInputs();
+        events.add(
+            runEvent(
+                terminal,
+                eventTypeOf(terminal),
+                propagateInputs ? withUpstreamInputs(ownInputs, transform, 
state) : ownInputs,
+                own == null ? List.of() : own.getOutputs(),
+                own == null ? null : own.getSoleSqlText(),
+                JobIdentity.of(ctx)
+                    .with(UPSTREAM_TRANSFORMS, previousTransforms(state, 
transform))));
+      } else {
+        events.add(runEvent(terminal, eventTypeOf(terminal), List.of(), 
List.of(), null));
+      }
+    }
+    for (Map.Entry<String, PendingRun> entry : state.transformIo.entrySet()) {
+      PendingRun own = entry.getValue();
+      if (!propagate) {
+        pipelineRun.absorb(own);
+      } else if (!finished.contains(entry.getKey()) && own.hasDatasets()) {
+        // A transform that touched datasets but never reported its end.
+        events.add(
+            standaloneEvent(
+                own,
+                propagateInputs
+                    ? withUpstreamInputs(own.getInputs(), entry.getKey(), 
state)
+                    : own.getInputs(),
+                System.currentTimeMillis()));
+      }
+    }
+    return events;
+  }
+
+  /**
+   * The datasets a transform's job consumes: its own reads plus everything 
its upstream transforms
+   * produced. That is what makes a dataset pair true — without it the job 
holds either the input or
+   * the output of an edge, never both.
+   */
+  private List<InputDataset> withUpstreamInputs(
+      List<InputDataset> own, String transformName, PipelineRunState state) {
+    List<InputDataset> combined = new ArrayList<>(own);
+    Set<DatasetRef> seen = new LinkedHashSet<>();
+    for (InputDataset input : own) {
+      seen.add(new DatasetRef(input.getNamespace(), input.getName()));
+    }
+    for (String upstream : previousTransforms(state, transformName)) {
+      for (DatasetRef ref : produced(state, upstream, new LinkedHashSet<>())) {
+        if (seen.add(ref)) {
+          combined.add(
+              ol.newInputDatasetBuilder()
+                  .namespace(ref.namespace())
+                  .name(ref.name())
+                  .facets(ol.newDatasetFacetsBuilder().build())
+                  .build());
+        }
+      }
+    }
+    return combined;
+  }
+
+  /**
+   * What a transform hands to its downstream transforms, in terms of 
datasets: what it wrote if it
+   * wrote anything (the stream carries the written content), otherwise what 
it read (a reader's
+   * product is the data it read), otherwise — a transform that does no I/O of 
its own, a filter or
+   * a select — whatever its own upstream transforms produced. That last case 
is what carries
+   * lineage through pass-through transforms.
+   */
+  private List<DatasetRef> produced(
+      PipelineRunState state, String transformName, Set<String> visited) {
+    if (Utils.isEmpty(transformName) || !visited.add(transformName)) {
+      return List.of();
+    }
+    PendingRun io = state.transformIo.get(transformName);
+    if (io != null && !io.getOutputs().isEmpty()) {
+      List<DatasetRef> produced = new ArrayList<>(io.getOutputs().size());
+      for (OutputDataset output : io.getOutputs()) {
+        produced.add(new DatasetRef(output.getNamespace(), output.getName()));
+      }
+      return produced;
+    }
+    if (io != null && !io.getInputs().isEmpty()) {
+      List<DatasetRef> produced = new ArrayList<>(io.getInputs().size());
+      for (InputDataset input : io.getInputs()) {
+        produced.add(new DatasetRef(input.getNamespace(), input.getName()));
+      }
+      return produced;
+    }
+    List<DatasetRef> produced = new ArrayList<>();
+    for (String upstream : previousTransforms(state, transformName)) {
+      produced.addAll(produced(state, upstream, visited));
+    }
+    return produced;
+  }
+
+  /**
+   * The transforms of a pipeline run that nothing in the run feeds: where its 
data enters. A
+   * collector that links the job that started the pipeline to the pipeline's 
first jobs uses them.
+   */
+  private static List<String> rootTransforms(PipelineRunState state) {
+    List<String> reported = new ArrayList<>();
+    for (LineageEvent terminal : state.deferredTerminals) {
+      reported.add(terminal.getContext().getTransformName());
+    }
+    return roots(state.graph, reported);
+  }
+
+  /** Every transform of a pipeline run: those in its hop graph and those that 
reported. */
+  private static List<String> allTransforms(PipelineRunState state) {
+    List<String> reported = new ArrayList<>();
+    for (LineageEvent terminal : state.deferredTerminals) {
+      reported.add(terminal.getContext().getTransformName());
+    }
+    return nodes(state.graph, reported);
+  }
+
+  /** The nodes of a hop graph, plus {@code others}, in order and once each. */
+  private static List<String> nodes(List<Transition> graph, List<String> 
others) {
+    Set<String> all = new LinkedHashSet<>();
+    for (Transition transition : graph) {
+      all.add(transition.getFrom());
+      all.add(transition.getTo());
+    }
+    all.addAll(others);
+    return new ArrayList<>(all);
+  }
+
+  /** The transforms that feed {@code transformName} in the run's hop graph. */
+  private static List<String> previousTransforms(PipelineRunState state, 
String transformName) {
+    return state.graph == null ? List.of() : leadingTo(state.graph, 
transformName);
+  }
+
+  /** The nodes of a hop graph (plus {@code others}) that no hop leads to. */
+  private static List<String> roots(List<Transition> graph, List<String> 
others) {
+    Set<String> led = new LinkedHashSet<>();
+    Set<String> all = new LinkedHashSet<>();
+    for (Transition transition : graph) {
+      all.add(transition.getFrom());
+      all.add(transition.getTo());
+      led.add(transition.getTo());
+    }
+    all.addAll(others);
+    all.removeAll(led);
+    return new ArrayList<>(all);
+  }
+
+  /** The nodes a hop leads from to {@code name}. */
+  private static List<String> leadingTo(List<Transition> graph, String name) {
+    List<String> previous = new ArrayList<>();
+    if (Utils.isEmpty(name)) {
+      return previous;
+    }
+    for (Transition transition : graph) {
+      if (name.equals(transition.getTo()) && 
!previous.contains(transition.getFrom())) {
+        previous.add(transition.getFrom());
+      }
+    }
+    return previous;
+  }
+
+  /** A dataset identity: what lineage edges are made of; facets are 
enrichment, not identity. */
+  private record DatasetRef(String namespace, String name) {}
+
+  /**
+   * What a run pushed out of the bounded state still had, as standalone 
events. A run is only
+   * pushed out when more than {@link #MAX_PENDING_RUNS} are in flight at 
once; its datasets are
+   * sent rather than dropped.
+   */
+  private List<RunEvent> drainEvicted() {
+    List<RunEvent> events = new ArrayList<>();
+    PendingRun run;
+    while ((run = evictedRuns.poll()) != null) {
+      if (run.hasDatasets()) {
+        events.add(standaloneEvent(run, run.getInputs(), 
System.currentTimeMillis()));
+      }
+    }
+    PipelineRunState state;
+    while ((state = evictedPipelineRuns.poll()) != null) {
+      events.addAll(abandon(state));
+    }
+    return events;
+  }
+
+  /** Everything a pipeline run that never reported its end collected, as 
events. */
+  private List<RunEvent> abandon(PipelineRunState state) {
+    PendingRun pipelineRun = new PendingRun(state.jobName, state.runId, 
state.identity);
+    List<RunEvent> events = finishPipelineRun(state, pipelineRun);
+    if (pipelineRun.hasDatasets()) {
+      events.add(standaloneEvent(pipelineRun, pipelineRun.getInputs(), 
System.currentTimeMillis()));
+    }
+    return events;
+  }
+
+  /**
+   * The observations still waiting for a run to end, as events, clearing 
them. The sink calls this
+   * when it shuts down: a run that never reported its end - an execution that 
crashed, or one that
+   * ran where the engine had no chance to say so - leaves nothing to carry 
its datasets, and
+   * dropping them would lose lineage.
+   */
+  public List<RunEvent> flushObservations() {
+    List<RunEvent> events = new ArrayList<>();
+    List<PipelineRunState> states;
+    synchronized (pipelineRuns) {
+      states = new ArrayList<>(pipelineRuns.values());
+      pipelineRuns.clear();
+    }
+    for (PipelineRunState state : states) {
+      events.addAll(abandon(state));
+    }
+    synchronized (pendingRuns) {
+      for (PendingRun run : pendingRuns.values()) {
+        if (run.hasDatasets()) {
+          events.add(standaloneEvent(run, run.getInputs(), 
System.currentTimeMillis()));
+        }
+      }
+      pendingRuns.clear();
+    }
+    events.addAll(drainEvicted());
+    return events;
+  }
+
+  /**
+   * A dataset observation as its own event, for the cases where no run event 
is left to carry it.
+   */
+  private RunEvent standaloneEvent(
+      PendingRun run, List<InputDataset> inputs, long timestampMillis) {
+    ZonedDateTime eventTime = toUtc(timestampMillis);
+    List<OutputDataset> outputs = run.getOutputs();
+    return ol.newRunEventBuilder()
+        .eventType(EventType.OTHER)
+        .eventTime(eventTime)
+        .run(runNode(run.getRunId(), eventTime, null))
+        .job(job(run.getJobName(), run.getIdentity(), run.getSoleSqlText()))
+        .inputs(inputs.isEmpty() ? null : inputs)
+        .outputs(outputs.isEmpty() ? null : outputs)
+        .build();
+  }
+
+  /**
+   * Identity of the run an observation belongs to, as the pair of names a 
collector sees for it:
+   * the job the sink names for the observation and the run id it uses for it. 
File and relational
+   * I/O is correlated to the pipeline run it happened in (see {@link 
#resolveFileIoRunId}), so a
+   * pipeline carries its transforms' datasets.
+   */
+  private static String runKey(String jobName, String runId) {
+    return jobName + '\u0000' + runId;
+  }
+
+  /** What a job is in Hop terms; see {@link #hopJobFacet}. */
+  record JobIdentity(
+      LineageSubjectType subjectType,
+      String pipelineName,
+      String workflowName,
+      String transformName,
+      String actionName,
+      String filename,
+      String description,
+      Map<String, Object> extra) {
+
+    /** The engine attributes that name who started a pipeline or workflow, 
copied to the facet. */
+    private static final List<String> PARENT_ATTRIBUTES =
+        List.of(
+            LineageRunLifecycleEmitter.ATTR_PARENT_WORKFLOW_NAME,
+            LineageRunLifecycleEmitter.ATTR_PARENT_ACTION_NAME,
+            LineageRunLifecycleEmitter.ATTR_PARENT_PIPELINE_NAME,
+            LineageRunLifecycleEmitter.ATTR_PARENT_TRANSFORM_NAME);
+
+    /** The subject the context describes. */
+    static JobIdentity of(LineageContext ctx) {
+      if (ctx == null) {
+        return new JobIdentity(null, null, null, null, null, null, null, 
Map.of());
+      }
+      Map<String, Object> extra = new LinkedHashMap<>();
+      if (ctx.getSubjectType() == LineageSubjectType.PIPELINE
+          || ctx.getSubjectType() == LineageSubjectType.WORKFLOW) {
+        for (String attribute : PARENT_ATTRIBUTES) {
+          String value = ctx.getAttributes().get(attribute);
+          if (!Utils.isEmpty(value)) {
+            extra.put(attribute, value);
+          }
+        }
+      }
+      return new JobIdentity(
+          ctx.getSubjectType(),
+          ctx.getPipelineName(),
+          ctx.getWorkflowName(),
+          ctx.getSubjectType() == LineageSubjectType.TRANSFORM ? 
ctx.getTransformName() : null,
+          ctx.getSubjectType() == LineageSubjectType.ACTION ? 
ctx.getActionName() : null,
+          filenameOf(ctx),
+          ctx.getAttributes().get(LineageRunLifecycleEmitter.ATTR_DESCRIPTION),
+          Collections.unmodifiableMap(extra));
+    }
+
+    /**
+     * The job a transform's I/O is filed under in the pipeline-granular 
mapping: its pipeline (see
+     * {@link #resolveFileIoJobName}).
+     */
+    static JobIdentity ioOwner(LineageContext ctx) {
+      if (ctx != null && !Utils.isEmpty(ctx.getPipelineName())) {
+        return new JobIdentity(
+            LineageSubjectType.PIPELINE,
+            ctx.getPipelineName(),
+            null,
+            null,
+            null,
+            filenameOf(ctx),
+            null,
+            Map.of());
+      }
+      return of(ctx);
+    }
+
+    /** This identity with one more facet property. */
+    JobIdentity with(String key, Object value) {
+      Map<String, Object> more = new LinkedHashMap<>(extra);
+      more.put(key, value);
+      return new JobIdentity(
+          subjectType,
+          pipelineName,
+          workflowName,
+          transformName,
+          actionName,
+          filename,
+          description,
+          Collections.unmodifiableMap(more));
+    }
+
+    private static String filenameOf(LineageContext ctx) {
+      return !Utils.isEmpty(ctx.getHopFilenamePortableKey())
+          ? ctx.getHopFilenamePortableKey()
+          : ctx.getHopFilename();
+    }
+  }
+
+  /**
+   * What a running pipeline run has collected with transform jobs on; see 
{@link #pipelineRuns}.
+   */
+  private static final class PipelineRunState {
+    private final String runId;
+    private final String jobName;
+    private final JobIdentity identity;
+    private List<Transition> graph;
+    private final Map<String, PendingRun> transformIo = new LinkedHashMap<>();
+    private final List<LineageEvent> deferredTerminals = new ArrayList<>();
+
+    PipelineRunState(String runId, String jobName, JobIdentity identity) {
+      this.runId = runId;
+      this.jobName = jobName;
+      this.identity = identity;
+    }
+  }
+
+  /** Datasets observed for one run, held until that run reports its terminal 
event. */
+  private static final class PendingRun {
+    private final String jobName;
+    private final String runId;
+    private final JobIdentity identity;
+    private final List<InputDataset> inputs = new ArrayList<>();
+    private final List<OutputDataset> outputs = new ArrayList<>();
+    private final Set<String> sqlTexts = new LinkedHashSet<>();
+
+    PendingRun(String jobName, String runId, JobIdentity identity) {
+      this.jobName = jobName;
+      this.runId = runId;
+      this.identity = identity;
+    }
+
+    String getJobName() {
+      return jobName;
+    }
+
+    String getRunId() {
+      return runId;
+    }
+
+    JobIdentity getIdentity() {
+      return identity;
+    }
+
+    List<InputDataset> getInputs() {
+      return inputs;
+    }
+
+    List<OutputDataset> getOutputs() {
+      return outputs;
+    }
+
+    /**
+     * The single statement a run executed, or {@code null} when it ran none 
or several: with
+     * several there is no one statement to claim the job ran.
+     */
+    String getSoleSqlText() {
+      return sqlTexts.size() == 1 ? sqlTexts.iterator().next() : null;
+    }
+
+    void add(List<InputDataset> newInputs, List<OutputDataset> newOutputs, 
String sqlText) {

Review Comment:
   `PendingRun.add` appends every observation's datasets without 
de-duplication, and observations are now held until the run's terminal event. A 
pipeline that loops over 100k files, or does a relational lookup per row, keeps 
100k InputDatasets in memory and produces a multi-MB COMPLETE event. If the 
collector answers 413, `failureOf(413)` is REJECTED, the spool file is deleted 
and the whole run's lineage (START/COMPLETE included) is lost. De-duplicating 
by dataset identity would keep this bounded.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to