hansva commented on code in PR #9339:
URL: https://github.com/apache/hop/pull/9339#discussion_r4536613185
##########
engine/src/main/java/org/apache/hop/lineage/hub/LineageHub.java:
##########
@@ -307,6 +315,9 @@ private void ensureWorkerStarted(LineageConfiguration cfg) {
thread.setDaemon(true);
worker = thread;
thread.start();
+ if (this == INSTANCE && exitHookRegistered.compareAndSet(false, true)) {
+ Runtime.getRuntime().addShutdownHook(new Thread(this::shutdown,
"Hop-LineageHub-Exit"));
Review Comment:
This exit hook calls `shutdown()` directly.
`HopEnvironment.initLifecycleListeners` already registers a hook that calls
`shutdownOnExit()`, which waits for runs that called `holdShutdownOnExit()`.
The JVM runs shutdown hooks concurrently, so on SIGTERM while a pipeline is
still stopping, this hook can shut the sinks down while the held run is still
emitting its terminal events, and those get lost (or start a new worker during
exit). I think this hook can be removed, or it should call `shutdownOnExit()`.
##########
engine/src/main/java/org/apache/hop/lineage/LineageActionStartListener.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage;
+
+import java.util.List;
+import org.apache.hop.core.Result;
+import org.apache.hop.workflow.IActionListener;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionMeta;
+import org.apache.hop.workflow.action.IAction;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+
+/**
+ * Emits an action's {@code STARTED} lineage event once the workflow has
cloned the action, so the
+ * event carries the run id of the action that actually executes. Registered
once per workflow run
+ * by {@link #register(IWorkflowEngine)}.
+ */
+public final class LineageActionStartListener implements
IActionListener<WorkflowMeta> {
+
+ private static final LineageActionStartListener INSTANCE = new
LineageActionStartListener();
+
+ private LineageActionStartListener() {}
+
+ /** Adds the listener to a workflow, unless it is already there (a workflow
can run again). */
+ @SuppressWarnings("unchecked")
+ public static void register(IWorkflowEngine<WorkflowMeta> workflow) {
+ if (workflow == null) {
+ return;
+ }
+ List<IActionListener> listeners = workflow.getActionListeners();
+ if (listeners == null || listeners.contains(INSTANCE)) {
+ return;
+ }
+ workflow.addActionListener(INSTANCE);
+ }
+
+ @Override
+ public void beforeExecution(
+ IWorkflowEngine<WorkflowMeta> workflow, ActionMeta actionMeta, IAction
action) {
+ LineageRunLifecycleEmitter.emitActionStarted(workflow, action);
Review Comment:
STARTED is now emitted from `IActionListener.beforeExecution`, inside
`Workflow.execute`'s critical section, without a try/catch. A RuntimeException
in `actionContextBuilder`, `fillFilenameFields` (e.g. variable resolution) or
`LineageHub.emit` now fails the action, the context classloader isn't restored,
and `releaseJoin(actionMeta)` is skipped (the finally only wraps
`cloneAction.execute()`). Could we wrap the emit in a try/catch and log, so a
lineage problem never fails the workflow?
##########
engine/src/main/java/org/apache/hop/lineage/xp/LineageHubPipelineCompletedXp.java:
##########
@@ -44,6 +44,9 @@ public void callExtensionPoint(
? RunLifecyclePhase.FAILED
: RunLifecyclePhase.FINISHED;
LineageRunLifecycleEmitter.emitPipeline(pipeline, phase, null);
- LineageHub.getInstance().flushQuietly();
+ // Only a top-level run waits for delivery; a nested run's events are
flushed by its parent.
+ if (LineageRunLifecycleEmitter.isTopLevel(pipeline)) {
Review Comment:
Nested pipelines/workflows no longer flush on completion. That flush was
effectively the only backpressure on the bounded lineage queue. A Pipeline
Executor running a child pipeline per row (say 100k rows) now queues events for
every child without waiting for the sink; once `HOP_LINEAGE_QUEUE_CAPACITY` is
reached, `emit()` drops events and logs an error for each one. Before, each
child completion blocked in `flushQuietly()`.
##########
engine/src/main/java/org/apache/hop/lineage/LineageRunLifecycleEmitter.java:
##########
@@ -55,8 +56,149 @@ public final class LineageRunLifecycleEmitter {
public static final String FAILED = "failed";
public static final String SUCCESS = "success";
+ /**
+ * The run that started this pipeline or workflow, when the engine knows it:
the action a workflow
+ * ran, or the parent pipeline of a sub-pipeline. {@code parentRunId} is
that subject's
+ * log-channel id and {@code parentJobName} is the job name the lineage
mapper gives it (e.g.
+ * {@code my-workflow/Run orders}); both are needed to form a
ParentRunFacet. Sinks that write the
+ * job tree themselves (DataHub) read the same pair.
+ */
+ public static final String ATTR_PARENT_RUN_ID = "parentRunId";
+
+ public static final String ATTR_PARENT_JOB_NAME = "parentJobName";
+
+ /**
+ * The parts {@link #ATTR_PARENT_JOB_NAME} is made of, for a sink that
models workflows and
+ * pipelines apart from the jobs inside them (DataHub): the workflow and
action that ran this run,
+ * or the pipeline and transform (Pipeline Executor, mapping) that did. A
job name cannot be split
+ * back reliably, since names may contain the separator.
+ */
+ public static final String ATTR_PARENT_WORKFLOW_NAME = "parentWorkflowName";
+
+ public static final String ATTR_PARENT_ACTION_NAME = "parentActionName";
+ public static final String ATTR_PARENT_PIPELINE_NAME = "parentPipelineName";
+ public static final String ATTR_PARENT_TRANSFORM_NAME =
"parentTransformName";
+
+ /** The description the subject's metadata gives it (pipeline, workflow,
transform, action). */
+ public static final String ATTR_DESCRIPTION = "description";
+
+ private static void fillDescription(LineageContext.Builder ctx, String
description) {
+ if (!Utils.isEmpty(description)) {
+ ctx.putAttribute(ATTR_DESCRIPTION, description);
+ }
+ }
+
private LineageRunLifecycleEmitter() {}
+ /**
+ * Records who started this run, when the engine knows: an action that ran a
pipeline or workflow,
+ * or a parent pipeline that ran a sub-pipeline. Left out entirely for a
top-level run.
+ */
+ private static void fillParentFields(
+ LineageContext.Builder ctx, String parentJobName, String parentRunId) {
+ if (!Utils.isEmpty(parentJobName)) {
+ ctx.putAttribute(ATTR_PARENT_JOB_NAME, parentJobName);
+ }
+ if (!Utils.isEmpty(parentRunId)) {
+ ctx.putAttribute(ATTR_PARENT_RUN_ID, parentRunId);
+ }
+ }
+
+ /**
+ * The run that started a pipeline: its sub-pipeline parent, or the workflow
action that ran it
+ * (with the workflow that owns the action). Package-private so the naming
rule can be checked
+ * without starting an engine.
+ */
+ static void fillPipelineParentFields(
+ LineageContext.Builder ctx, IPipelineEngine<PipelineMeta> pipeline) {
+ IPipelineEngine<PipelineMeta> parentPipeline =
pipeline.getParentPipeline();
+ if (parentPipeline != null) {
+ fillPipelineParentFields(ctx, parentPipeline, pipeline.getParent());
+ return;
+ }
+ if (pipeline.getParent() instanceof IAction parentAction) {
+ fillActionParentFields(ctx, parentAction, pipeline.getParentWorkflow());
+ }
+ }
+
+ /** A run started from a pipeline, by the transform that is its logging
parent when known. */
+ private static void fillPipelineParentFields(
+ LineageContext.Builder ctx, IPipelineEngine<PipelineMeta>
parentPipeline, Object parent) {
+ PipelineMeta parentMeta = parentPipeline.getPipelineMeta();
+ String parentPipelineName = parentMeta != null ? parentMeta.getName() :
null;
+ fillParentFields(ctx, parentPipelineName,
parentPipeline.getLogChannelId());
Review Comment:
Parent attribution differs per launch type: a pipeline started by a workflow
action gets `<workflow>/<action>` with the action's run id, but a sub-pipeline
from a Pipeline Executor or mapping only gets the parent pipeline's name and
log channel, not `<pipeline>/<transform>`. The comment in `emitPipeline` says
both follow the `<owner>/<job>` convention, but in OpenLineage the
ParentRunFacet of an executor child points at the whole parent pipeline job, so
sinks get two different tree shapes for the same relationship.
##########
engine/src/main/java/org/apache/hop/lineage/LineageActionStartListener.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage;
+
+import java.util.List;
+import org.apache.hop.core.Result;
+import org.apache.hop.workflow.IActionListener;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionMeta;
+import org.apache.hop.workflow.action.IAction;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+
+/**
+ * Emits an action's {@code STARTED} lineage event once the workflow has
cloned the action, so the
+ * event carries the run id of the action that actually executes. Registered
once per workflow run
+ * by {@link #register(IWorkflowEngine)}.
+ */
+public final class LineageActionStartListener implements
IActionListener<WorkflowMeta> {
+
+ private static final LineageActionStartListener INSTANCE = new
LineageActionStartListener();
+
+ private LineageActionStartListener() {}
+
+ /** Adds the listener to a workflow, unless it is already there (a workflow
can run again). */
+ @SuppressWarnings("unchecked")
+ public static void register(IWorkflowEngine<WorkflowMeta> workflow) {
+ if (workflow == null) {
+ return;
+ }
+ List<IActionListener> listeners = workflow.getActionListeners();
+ if (listeners == null || listeners.contains(INSTANCE)) {
+ return;
+ }
+ workflow.addActionListener(INSTANCE);
+ }
+
+ @Override
+ public void beforeExecution(
+ IWorkflowEngine<WorkflowMeta> workflow, ActionMeta actionMeta, IAction
action) {
+ LineageRunLifecycleEmitter.emitActionStarted(workflow, action);
+ }
+
+ @Override
+ public void afterExecution(
+ IWorkflowEngine<WorkflowMeta> workflow,
+ ActionMeta actionMeta,
+ IAction action,
+ Result result) {
+ // FINISHED and FAILED come from the WorkflowAfterActionExecution
extension point, which also
Review Comment:
I don't think this holds: if `cloneAction.execute()` throws,
`Workflow.execute` never reaches the `WorkflowAfterActionExecution` extension
point. STARTED has been emitted, but no FINISHED/FAILED ever follows for that
action run id, so sinks keep the run open forever.
##########
plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInput.java:
##########
@@ -663,4 +665,14 @@ protected IBaseFileInputReader createReader(
JsonInputMeta meta, JsonInputData data, FileObject file) throws
Exception {
throw new NotImplementedException();
}
+
+ /** The fields this transform reads from the file; added and passed-through
fields are not. */
+ @Override
+ protected Set<String> fileFieldNames() {
Review Comment:
As far as I can see this override is never called: JsonInput doesn't go
through `emitLineageFileRead` (`createReader` throws `NotImplementedException`)
and emits its own FILE_IO events with `jsonFileReadContentSchema()`. Could it
be dropped, so there's one description of the JSON file schema?
##########
engine/src/main/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransform.java:
##########
@@ -175,8 +178,41 @@ protected void emitLineageFileRead(FileObject file) {
} catch (Exception ignored) {
// optional metadata for lineage only
}
+ String extension = file.getName().getExtension();
LineageFileIoEmitter.emitTransformFileIo(
- this, FileIoOperation.READ, file, null, size, true, null);
+ this,
+ FileIoOperation.READ,
+ file,
+ null,
+ size,
+ true,
+ null,
+ LineageFileIoEmitter.toContentSchema(
Review Comment:
`toContentSchema(fileRowMeta(), ...)` is evaluated as an argument before
`emitTransformFileIo` checks `isEnabled()`, so every file open builds a
HashSet, a filtered RowMeta and a column list even with lineage off. For a Text
File Input reading thousands of small files that's all wasted. Maybe return
early in `emitLineageFileRead` when `!LineageFileIoEmitter.isEnabled()`?
##########
plugins/transforms/textfile/src/main/java/org/apache/hop/pipeline/transforms/fileinput/text/TextFileInput.java:
##########
@@ -623,4 +625,14 @@ private void rejectCurrentFile(String errorMsg) {
}
}
}
+
+ /** The fields this transform reads from the file; added and passed-through
fields are not. */
+ @Override
+ protected Set<String> fileFieldNames() {
Review Comment:
`fileFieldNames()` is nearly identical here and in `JsonInput`, and it's
rebuilt on every file open; it could be computed once (e.g. in
`prepareToRowProcessing`) in the base class. Also, a pass-through input field
with the same name as a file field passes this filter and ends up in the file's
schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]