This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 3c9513615e Beam Flink pipeline engine page: Flink 1.19, one Flink job
ID section, no Javadoc leftovers. #9048 (#9222)
3c9513615e is described below
commit 3c9513615ed2a743f3d0088ac9ee6e714913025e
Author: Adalennis <[email protected]>
AuthorDate: Tue Nov 3 09:01:52 2026 +0100
Beam Flink pipeline engine page: Flink 1.19, one Flink job ID section, no
Javadoc leftovers. #9048 (#9222)
---
.../beam-flink-pipeline-engine.adoc | 23 +++++++++++-----------
1 file changed, 11 insertions(+), 12 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
index c6610c74ba..dce4a9944f 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc
@@ -22,7 +22,7 @@ under the License.
== Beam Flink
-This runner allows you to run Hop pipelines on https://flink.apache.org[Apache
Flink] version 1.13.
+This runner allows you to run Hop pipelines on https://flink.apache.org[Apache
Flink] version 1.19. See
xref:pipeline/beam/getting-started-with-beam.adoc[Getting started with Apache
Beam] for the Beam and Flink versions of each Hop release.
The Flink runner supports two modes: Local Direct Flink Runner and Flink
Runner.
@@ -60,7 +60,7 @@ If this is set to false, the task will only decline a the
checkpoint and continu
A value of zero effectively disables fault tolerance.
A value of -1 indicates that the system default value (as defined in the
configuration) should be used.|-1
|Execution retry delay (ms)|Sets the delay in milliseconds between executions.
-A value of {@code -1} indicates that the default value should be used.|-1
+A value of -1 indicates that the default value should be used.|-1
|Object re-use|Sets the behavior of reusing objects.
Enabling the object reuse mode will instruct the runtime to reuse user objects
for better performance.|false
|Disable metrics|Disable Beam metrics in Flink Runner|-1
@@ -76,7 +76,7 @@ See
https://issues.apache.org/jira/browse/FLINK-2491[FLINK-2491] for progress on
Interval value <= 0 disables the feature.|0
|Auto watermark interval|The interval in milliseconds for automatic watermark
emission.|
|Batch execution mode|Flink mode for data exchange of batch pipelines.
-Reference {@link org.apache.flink.api.common.ExecutionMode}.
+The values are those of Flink's `org.apache.flink.api.common.ExecutionMode`.
Set this to BATCH_FORCED if pipelines get blocked, see
https://issues.apache.org/jira/browse/FLINK-10672[FLINK-10672]|P
|User agent|A user agent string as per
https://tools.ietf.org/html/rfc2616[RFC2616], describing the pipeline to
external services.|
|Temp location|Path for temporary files.|
@@ -88,9 +88,10 @@ Set this to BATCH_FORCED if pipelines get blocked, see
https://issues.apache.org
|Fat jar file location|Fat jar location.|
|===
-== Job id
+== Flink job ID
-Beam does not return the Flink job id on the pipeline result. Hop chooses the
id before submitting the job. After `prepareExecution()` it is available from
the engine:
+Beam doesn't return the Flink job ID on the pipeline result, so Hop chooses
the ID itself before it submits the job, when the Flink master is `[local]` or
`host:port`.
+After `prepareExecution()` it is available from the engine:
[source,java]
----
@@ -99,7 +100,11 @@ if (pipeline instanceof BeamFlinkPipelineEngine
flinkEngine) {
}
----
-The same value is written to the pipeline log. When the run configuration has
an execution information location, it is stored as `flink.job.id` and shown on
the execution viewer info tab. A master of `[collection]` does not submit a
cluster job, so no id is assigned.
+The same value is written to the pipeline log. When the run configuration has
an execution information location, it is stored as `flink.job.id` and shown on
the execution viewer info tab.
+
+With a master of `[auto]`, which is also used when the master is empty, Flink
takes its environment from `flink run` or the Kubernetes operator and ignores
the ID Hop would set, and a master of `[collection]` doesn't submit a cluster
job: in those cases Hop has no Flink job ID.
+Use the Beam job name instead: Hop derives it from the pipeline name, and you
can find the job by that name in the Flink UI or with the Flink REST API.
+The Dataflow runner is different: Hop reports that job ID through the
execution detail `dataflow.job.id`.
== Running with Flink Run
@@ -170,9 +175,3 @@ bin/flink run \
You can specify a master of `[local]` to run using an embedded Flink engine.
It's primarily used for testing locally.
-
-== Flink job ID
-
-Hop does not show a Flink job ID after a Beam run. Beam 2.74 keeps the
attached result as accumulators and a runtime, and the detached result keeps
its job client in a private field with no accessor. There is no supported way
to read the job ID from `PipelineResult`.
-
-The Beam job name is still available. Hop derives it from the pipeline name.
On a Flink cluster, use that name in the Flink UI or the Flink REST API to find
the job. The Dataflow runner is different: Hop reports that job ID through the
execution detail `dataflow.job.id`.