hansva commented on code in PR #9339:
URL: https://github.com/apache/hop/pull/9339#discussion_r4536613185


##########
engine/src/main/java/org/apache/hop/lineage/hub/LineageHub.java:
##########
@@ -307,6 +315,9 @@ private void ensureWorkerStarted(LineageConfiguration cfg) {
       thread.setDaemon(true);
       worker = thread;
       thread.start();
+      if (this == INSTANCE && exitHookRegistered.compareAndSet(false, true)) {
+        Runtime.getRuntime().addShutdownHook(new Thread(this::shutdown, 
"Hop-LineageHub-Exit"));

Review Comment:
   This exit hook calls `shutdown()` directly. 
`HopEnvironment.initLifecycleListeners` already registers a hook that calls 
`shutdownOnExit()`, which waits for runs that called `holdShutdownOnExit()`. 
The JVM runs shutdown hooks concurrently, so on SIGTERM while a pipeline is 
still stopping, this hook can shut the sinks down while the held run is still 
emitting its terminal events, and those get lost (or start a new worker during 
exit). I think this hook can be removed, or it should call `shutdownOnExit()`.



##########
engine/src/main/java/org/apache/hop/lineage/LineageActionStartListener.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage;
+
+import java.util.List;
+import org.apache.hop.core.Result;
+import org.apache.hop.workflow.IActionListener;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionMeta;
+import org.apache.hop.workflow.action.IAction;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+
+/**
+ * Emits an action's {@code STARTED} lineage event once the workflow has 
cloned the action, so the
+ * event carries the run id of the action that actually executes. Registered 
once per workflow run
+ * by {@link #register(IWorkflowEngine)}.
+ */
+public final class LineageActionStartListener implements 
IActionListener<WorkflowMeta> {
+
+  private static final LineageActionStartListener INSTANCE = new 
LineageActionStartListener();
+
+  private LineageActionStartListener() {}
+
+  /** Adds the listener to a workflow, unless it is already there (a workflow 
can run again). */
+  @SuppressWarnings("unchecked")
+  public static void register(IWorkflowEngine<WorkflowMeta> workflow) {
+    if (workflow == null) {
+      return;
+    }
+    List<IActionListener> listeners = workflow.getActionListeners();
+    if (listeners == null || listeners.contains(INSTANCE)) {
+      return;
+    }
+    workflow.addActionListener(INSTANCE);
+  }
+
+  @Override
+  public void beforeExecution(
+      IWorkflowEngine<WorkflowMeta> workflow, ActionMeta actionMeta, IAction 
action) {
+    LineageRunLifecycleEmitter.emitActionStarted(workflow, action);

Review Comment:
   STARTED is now emitted from `IActionListener.beforeExecution`, inside 
`Workflow.execute`'s critical section, without a try/catch. A RuntimeException 
in `actionContextBuilder`, `fillFilenameFields` (e.g. variable resolution) or 
`LineageHub.emit` now fails the action, the context classloader isn't restored, 
and `releaseJoin(actionMeta)` is skipped (the finally only wraps 
`cloneAction.execute()`). Could we wrap the emit in a try/catch and log, so a 
lineage problem never fails the workflow?



##########
engine/src/main/java/org/apache/hop/lineage/xp/LineageHubPipelineCompletedXp.java:
##########
@@ -44,6 +44,9 @@ public void callExtensionPoint(
             ? RunLifecyclePhase.FAILED
             : RunLifecyclePhase.FINISHED;
     LineageRunLifecycleEmitter.emitPipeline(pipeline, phase, null);
-    LineageHub.getInstance().flushQuietly();
+    // Only a top-level run waits for delivery; a nested run's events are 
flushed by its parent.
+    if (LineageRunLifecycleEmitter.isTopLevel(pipeline)) {

Review Comment:
   Nested pipelines/workflows no longer flush on completion. That flush was 
effectively the only backpressure on the bounded lineage queue. A Pipeline 
Executor running a child pipeline per row (say 100k rows) now queues events for 
every child without waiting for the sink; once `HOP_LINEAGE_QUEUE_CAPACITY` is 
reached, `emit()` drops events and logs an error for each one. Before, each 
child completion blocked in `flushQuietly()`.



##########
engine/src/main/java/org/apache/hop/lineage/LineageRunLifecycleEmitter.java:
##########
@@ -55,8 +56,149 @@ public final class LineageRunLifecycleEmitter {
   public static final String FAILED = "failed";
   public static final String SUCCESS = "success";
 
+  /**
+   * The run that started this pipeline or workflow, when the engine knows it: 
the action a workflow
+   * ran, or the parent pipeline of a sub-pipeline. {@code parentRunId} is 
that subject's
+   * log-channel id and {@code parentJobName} is the job name the lineage 
mapper gives it (e.g.
+   * {@code my-workflow/Run orders}); both are needed to form a 
ParentRunFacet. Sinks that write the
+   * job tree themselves (DataHub) read the same pair.
+   */
+  public static final String ATTR_PARENT_RUN_ID = "parentRunId";
+
+  public static final String ATTR_PARENT_JOB_NAME = "parentJobName";
+
+  /**
+   * The parts {@link #ATTR_PARENT_JOB_NAME} is made of, for a sink that 
models workflows and
+   * pipelines apart from the jobs inside them (DataHub): the workflow and 
action that ran this run,
+   * or the pipeline and transform (Pipeline Executor, mapping) that did. A 
job name cannot be split
+   * back reliably, since names may contain the separator.
+   */
+  public static final String ATTR_PARENT_WORKFLOW_NAME = "parentWorkflowName";
+
+  public static final String ATTR_PARENT_ACTION_NAME = "parentActionName";
+  public static final String ATTR_PARENT_PIPELINE_NAME = "parentPipelineName";
+  public static final String ATTR_PARENT_TRANSFORM_NAME = 
"parentTransformName";
+
+  /** The description the subject's metadata gives it (pipeline, workflow, 
transform, action). */
+  public static final String ATTR_DESCRIPTION = "description";
+
+  private static void fillDescription(LineageContext.Builder ctx, String 
description) {
+    if (!Utils.isEmpty(description)) {
+      ctx.putAttribute(ATTR_DESCRIPTION, description);
+    }
+  }
+
   private LineageRunLifecycleEmitter() {}
 
+  /**
+   * Records who started this run, when the engine knows: an action that ran a 
pipeline or workflow,
+   * or a parent pipeline that ran a sub-pipeline. Left out entirely for a 
top-level run.
+   */
+  private static void fillParentFields(
+      LineageContext.Builder ctx, String parentJobName, String parentRunId) {
+    if (!Utils.isEmpty(parentJobName)) {
+      ctx.putAttribute(ATTR_PARENT_JOB_NAME, parentJobName);
+    }
+    if (!Utils.isEmpty(parentRunId)) {
+      ctx.putAttribute(ATTR_PARENT_RUN_ID, parentRunId);
+    }
+  }
+
+  /**
+   * The run that started a pipeline: its sub-pipeline parent, or the workflow 
action that ran it
+   * (with the workflow that owns the action). Package-private so the naming 
rule can be checked
+   * without starting an engine.
+   */
+  static void fillPipelineParentFields(
+      LineageContext.Builder ctx, IPipelineEngine<PipelineMeta> pipeline) {
+    IPipelineEngine<PipelineMeta> parentPipeline = 
pipeline.getParentPipeline();
+    if (parentPipeline != null) {
+      fillPipelineParentFields(ctx, parentPipeline, pipeline.getParent());
+      return;
+    }
+    if (pipeline.getParent() instanceof IAction parentAction) {
+      fillActionParentFields(ctx, parentAction, pipeline.getParentWorkflow());
+    }
+  }
+
+  /** A run started from a pipeline, by the transform that is its logging 
parent when known. */
+  private static void fillPipelineParentFields(
+      LineageContext.Builder ctx, IPipelineEngine<PipelineMeta> 
parentPipeline, Object parent) {
+    PipelineMeta parentMeta = parentPipeline.getPipelineMeta();
+    String parentPipelineName = parentMeta != null ? parentMeta.getName() : 
null;
+    fillParentFields(ctx, parentPipelineName, 
parentPipeline.getLogChannelId());

Review Comment:
   Parent attribution differs per launch type: a pipeline started by a workflow 
action gets `<workflow>/<action>` with the action's run id, but a sub-pipeline 
from a Pipeline Executor or mapping only gets the parent pipeline's name and 
log channel, not `<pipeline>/<transform>`. The comment in `emitPipeline` says 
both follow the `<owner>/<job>` convention, but in OpenLineage the 
ParentRunFacet of an executor child points at the whole parent pipeline job, so 
sinks get two different tree shapes for the same relationship.



##########
engine/src/main/java/org/apache/hop/lineage/LineageActionStartListener.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage;
+
+import java.util.List;
+import org.apache.hop.core.Result;
+import org.apache.hop.workflow.IActionListener;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionMeta;
+import org.apache.hop.workflow.action.IAction;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+
+/**
+ * Emits an action's {@code STARTED} lineage event once the workflow has 
cloned the action, so the
+ * event carries the run id of the action that actually executes. Registered 
once per workflow run
+ * by {@link #register(IWorkflowEngine)}.
+ */
+public final class LineageActionStartListener implements 
IActionListener<WorkflowMeta> {
+
+  private static final LineageActionStartListener INSTANCE = new 
LineageActionStartListener();
+
+  private LineageActionStartListener() {}
+
+  /** Adds the listener to a workflow, unless it is already there (a workflow 
can run again). */
+  @SuppressWarnings("unchecked")
+  public static void register(IWorkflowEngine<WorkflowMeta> workflow) {
+    if (workflow == null) {
+      return;
+    }
+    List<IActionListener> listeners = workflow.getActionListeners();
+    if (listeners == null || listeners.contains(INSTANCE)) {
+      return;
+    }
+    workflow.addActionListener(INSTANCE);
+  }
+
+  @Override
+  public void beforeExecution(
+      IWorkflowEngine<WorkflowMeta> workflow, ActionMeta actionMeta, IAction 
action) {
+    LineageRunLifecycleEmitter.emitActionStarted(workflow, action);
+  }
+
+  @Override
+  public void afterExecution(
+      IWorkflowEngine<WorkflowMeta> workflow,
+      ActionMeta actionMeta,
+      IAction action,
+      Result result) {
+    // FINISHED and FAILED come from the WorkflowAfterActionExecution 
extension point, which also

Review Comment:
   I don't think this holds: if `cloneAction.execute()` throws, 
`Workflow.execute` never reaches the `WorkflowAfterActionExecution` extension 
point. STARTED has been emitted, but no FINISHED/FAILED ever follows for that 
action run id, so sinks keep the run open forever.



##########
plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInput.java:
##########
@@ -663,4 +665,14 @@ protected IBaseFileInputReader createReader(
       JsonInputMeta meta, JsonInputData data, FileObject file) throws 
Exception {
     throw new NotImplementedException();
   }
+
+  /** The fields this transform reads from the file; added and passed-through 
fields are not. */
+  @Override
+  protected Set<String> fileFieldNames() {

Review Comment:
   As far as I can see this override is never called: JsonInput doesn't go 
through `emitLineageFileRead` (`createReader` throws `NotImplementedException`) 
and emits its own FILE_IO events with `jsonFileReadContentSchema()`. Could it 
be dropped, so there's one description of the JSON file schema?



##########
engine/src/main/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransform.java:
##########
@@ -175,8 +178,41 @@ protected void emitLineageFileRead(FileObject file) {
     } catch (Exception ignored) {
       // optional metadata for lineage only
     }
+    String extension = file.getName().getExtension();
     LineageFileIoEmitter.emitTransformFileIo(
-        this, FileIoOperation.READ, file, null, size, true, null);
+        this,
+        FileIoOperation.READ,
+        file,
+        null,
+        size,
+        true,
+        null,
+        LineageFileIoEmitter.toContentSchema(

Review Comment:
   `toContentSchema(fileRowMeta(), ...)` is evaluated as an argument before 
`emitTransformFileIo` checks `isEnabled()`, so every file open builds a 
HashSet, a filtered RowMeta and a column list even with lineage off. For a Text 
File Input reading thousands of small files that's all wasted. Maybe return 
early in `emitLineageFileRead` when `!LineageFileIoEmitter.isEnabled()`?



##########
plugins/transforms/textfile/src/main/java/org/apache/hop/pipeline/transforms/fileinput/text/TextFileInput.java:
##########
@@ -623,4 +625,14 @@ private void rejectCurrentFile(String errorMsg) {
       }
     }
   }
+
+  /** The fields this transform reads from the file; added and passed-through 
fields are not. */
+  @Override
+  protected Set<String> fileFieldNames() {

Review Comment:
   `fileFieldNames()` is nearly identical here and in `JsonInput`, and it's 
rebuilt on every file open; it could be computed once (e.g. in 
`prepareToRowProcessing`) in the base class. Also, a pass-through input field 
with the same name as a file field passes this filter and ends up in the file's 
schema.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to