This is an automated email from the ASF dual-hosted git repository.
bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new e6d9ef48f6 Issue #2632 : Document scheduling options for workflows and
pipelines (#8273)
e6d9ef48f6 is described below
commit e6d9ef48f6ebc834d57b7e6d59b9dfeb08a748b3
Author: Matt Casters <[email protected]>
AuthorDate: Tue Sep 8 09:05:42 2026 +0200
Issue #2632 : Document scheduling options for workflows and pipelines
(#8273)
* Issue #2632 : Document scheduling options for workflows and pipelines
Add a how-to that shows how to start hop-run or a short-lived container
from cron, at, systemd, Windows Task Scheduler, Jenkins, Kubernetes
CronJob, and Apache Airflow, and link it from the existing docs.
* Issue #2632 : Fix cron mail claim and Windows call in scheduling how-to
cron mails on any stdout, not on a non-zero exit, so the wrapper now
discards console output after -lf. The Windows wrapper uses call so
control returns to the .cmd file.
---
docs/hop-user-manual/modules/ROOT/nav.adoc | 1 +
docs/hop-user-manual/modules/ROOT/pages/cloud.adoc | 4 +-
.../modules/ROOT/pages/docker-container.adoc | 2 +-
.../ROOT/pages/getting-started/hop-next-steps.adoc | 1 +
.../modules/ROOT/pages/hop-run/index.adoc | 5 +
.../pages/hop-server/deploy-project-image.adoc | 1 +
.../modules/ROOT/pages/hop-server/deploying.adoc | 5 +-
.../modules/ROOT/pages/how-to-guides/index.adoc | 3 +-
.../how-to-guides/run-hop-in-apache-airflow.adoc | 8 +
.../scheduling-workflows-and-pipelines.adoc | 470 +++++++++++++++++++++
.../ROOT/pages/pipeline/pipeline-unit-testing.adoc | 2 +-
.../pages/snippets/best-practices/governance.adoc | 1 +
.../ROOT/pages/workflow/actions/fileexists.adoc | 2 +-
.../modules/ROOT/pages/workflow/actions/start.adoc | 2 +-
14 files changed, 499 insertions(+), 8 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc
b/docs/hop-user-manual/modules/ROOT/nav.adoc
index 33873a8900..a2d01fd56e 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -589,6 +589,7 @@ under the License.
** xref:how-to-guides/logging-workflow-log.adoc[Logging workflow data with
workflow log]
** xref:how-to-guides/loops-in-apache-hop.adoc[Loops in Apache Hop]
** xref:how-to-guides/workflows-parallel-execution.adoc[Parallel execution in
workflows]
+** xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines]
** xref:how-to-guides/run-hop-in-apache-airflow.adoc[Run Hop workflows and
pipelines in Apache Airflow]
** xref:how-to-guides/avoiding-deadlocks.adoc[Avoiding deadlocks]
* xref:community-blogs/index.adoc[Community Posts]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/cloud.adoc
b/docs/hop-user-manual/modules/ROOT/pages/cloud.adoc
index 3f666986d2..d99a22d08a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/cloud.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/cloud.adoc
@@ -153,7 +153,8 @@ Fire-and-forget submission to a Hop Server needs you to
poll or to use the xref:
Hop Server does not keep a durable work queue; a restart drops in-flight
executions unless `--shutdown-timeout` / `HOP_SERVER_SHUTDOWN_TIMEOUT` is set
and the platform honours it during a graceful shutdown.
Short-lived containers do not have this problem: the orchestrator retries the
job.
-The xref:how-to-guides/run-hop-in-apache-airflow.adoc[Apache Airflow how-to]
is a worked example of the pattern; the same principle applies to any scheduler.
+The xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[scheduling
how-to] is the catalogue of those options (cron, systemd, Jenkins, a Kubernetes
CronJob, Airflow, ...).
+The xref:how-to-guides/run-hop-in-apache-airflow.adoc[Apache Airflow how-to]
is a worked example of one of them; the same principle applies to any scheduler.
== 5. Where run history and logs go
@@ -258,5 +259,6 @@ If your provider is not in these lists, the question to ask
is which standard it
* xref:hop-server/deploying.adoc[Deploying Hop Server] — the three ways a
project reaches a server, compared
* xref:projects/index.adoc[Projects and environments] — what belongs in the
project and what belongs in the environment
* xref:metadata-types/execution-information-location.adoc[Execution
Information Location] — where run history is kept
+* xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines] — cron, systemd, Jenkins, Kubernetes CronJob, Airflow
* xref:how-to-guides/run-hop-in-apache-airflow.adoc[Run Hop workflows and
pipelines in Apache Airflow] — a worked scheduler example
* xref:best-practices/index.adoc[Best Practices]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc
b/docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc
index 1674404533..7a02f4f362 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc
@@ -287,7 +287,7 @@ That image will contain the last built Development snapshot
of Apache Hop. rxq77
The most common use case will be that you run a **short-lived container** to
just complete one Hop workflow or pipeline.
If CI already baked the project into the image, this is often all you need in
production: a scheduler starts the container, hop-run finishes, the container
exits -- no long-lived Hop Server.
-See
xref:hop-server/deploy-project-image.adoc#skip_hop_server_short_lived_containers[Skip
Hop Server: short-lived containers].
+See
xref:hop-server/deploy-project-image.adoc#skip_hop_server_short_lived_containers[Skip
Hop Server: short-lived containers] and
xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling workflows
and pipelines].
The first example below runs the sample `switch-case-basic.hpl` **pipeline**
from the samples project.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
index c98c1a5e3d..1867d6699a 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
@@ -33,6 +33,7 @@ Here are a couple of topics you may want to look into:
* xref:how-to-guides/change-data-capture.adoc[Change Data Capture] covers the
three ways Hop detects inserts, updates and deletes: log sniffing, filtered
selection, and snapshot comparison.
* xref:best-practices/index.adoc[Best Practices] covers a number of things you
might want to think about while using Apache Hop.
* xref:projects/index.adoc[Projects] explains how to work with projects and
environments
+* xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines] shows how to start a run from cron, systemd, Jenkins,
a Kubernetes CronJob or Apache Airflow
* xref:hop-server/deploying.adoc[Deploying Hop Server] covers the three common
ways to get a project onto a running server
* xref:vfs.adoc[VFS] explains how you can access resources in the 3 main cloud
platforms: AWS, Azure and GCP.
* xref:logging/logging-basics.adoc[Logging] explains how to configure Hop for
your desired log level and target platform
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-run/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-run/index.adoc
index b313f6eaa3..a901a9ae59 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-run/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-run/index.adoc
@@ -289,3 +289,8 @@ Result:
|Key|value
|key1|"String with spaces"
|===
+
+== Scheduling
+
+Hop Run is the command a scheduler should invoke: it runs one file and exits.
+See xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines] for cron, `at`, systemd, Windows Task Scheduler,
Jenkins, Kubernetes CronJob and Apache Airflow.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-project-image.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-project-image.adoc
index 3e7b575045..0a0fdf16e9 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-project-image.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-project-image.adoc
@@ -186,6 +186,7 @@ docker run --rm \
Airflow can do this with the DockerOperator (or KubernetesPodOperator on a
cluster).
That is exactly the xref:how-to-guides/run-hop-in-apache-airflow.adoc[Airflow
how-to], except the image already contains `/your-project` so you do not mount
the project from the worker.
On Kubernetes or OpenShift the same idea is a Job or CronJob whose container
spec is this image plus those environment variables.
+See xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines] for a CronJob manifest, and for cron, systemd and
Jenkins starting the same image.
The container is gone when hop-run finishes, so anything you still want to
look at later must be written *outside* it.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploying.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploying.adoc
index 9982d7c2e5..15c5095fc8 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploying.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploying.adoc
@@ -24,6 +24,7 @@ under the License.
xref:hop-server/index.adoc[Hop Server] is a long-lived process that executes
pipelines and workflows.
It is not a scheduler: something else (Hop Gui,
xref:hop-run/index.adoc[hop-run], Apache Airflow, cron, Jenkins, a Kubernetes
CronJob, ...) has to submit the work.
+See xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines].
This page is the deployment story: after you develop a project, how does a
running Hop Server get the files and the configuration it needs?
@@ -122,7 +123,7 @@ You can inspect those runs later from the
xref:hop-gui/perspective-execution-inf
This is the usual production pattern when a scheduler (Airflow, Jenkins, cron,
a Kubernetes Job or CronJob) already owns the calendar.
Keep a long-lived Hop Server when you need
xref:hop-server/web-service.adoc[web services], or when many clients submit
work to a shared always-on engine.
-See
xref:hop-server/deploy-project-image.adoc#skip_hop_server_short_lived_containers[Skip
Hop Server: short-lived containers] and the
xref:how-to-guides/run-hop-in-apache-airflow.adoc[Airflow] how-to.
+See
xref:hop-server/deploy-project-image.adoc#skip_hop_server_short_lived_containers[Skip
Hop Server: short-lived containers],
xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling workflows
and pipelines], and the
xref:how-to-guides/run-hop-in-apache-airflow.adoc[Airflow] how-to.
== Environment configuration
@@ -167,7 +168,7 @@ The server does not read them.
== What Hop Server does not do
* It does not schedule work.
-Use Airflow, cron, Jenkins, a Kubernetes CronJob, or hop-run from a wrapper.
+Use xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[an external
scheduler]: Airflow, cron, Jenkins, a Kubernetes CronJob, or hop-run from a
wrapper.
* It does not keep a durable work queue.
A restart drops in-flight executions unless you set `--shutdown-timeout` /
`HOP_SERVER_SHUTDOWN_TIMEOUT` and the orchestrator waits.
* A short-lived `apache/hop` container that runs `hop-run` and exits is *not*
Hop Server.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
index 59cc499a06..698a680af6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
@@ -14,7 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-[[HopServer]]
+[[HowToGuides]]
:imagesdir: ../../assets/images
:description: This page contains a collection of how-to guides to perform a
variety of tasks, configurations etc with Apache Hop.
@@ -28,6 +28,7 @@ This page contains a collection of how-to guides to perform a
variety of tasks,
** xref:how-to-guides/cdc-snapshot-comparison.adoc[CDC: comparing snapshots]
* xref:how-to-guides/joins-lookups.adoc[Joins and lookups in Apache Hop]
* xref:how-to-guides/loops-in-apache-hop.adoc[Loops in Apache Hop]
+* xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines]
* xref:how-to-guides/run-hop-in-apache-airflow.adoc[Run Pipelines and
Workflows from Apache Airflow]
* xref:how-to-guides/apache-hop-web-services-docker.adoc[Using Apache Hop web
services in Docker]
* xref:hop-server/deploying.adoc[Deploying Hop Server]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/run-hop-in-apache-airflow.adoc
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/run-hop-in-apache-airflow.adoc
index a0d3ce115b..663c3335b2 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/run-hop-in-apache-airflow.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/run-hop-in-apache-airflow.adoc
@@ -20,6 +20,9 @@ under the License.
= image:how-to-guides/run-hop-in-apache-airflow/airflow-logo.svg[Apache
Airflow, width="75vw", align="center"]Run workflows and pipelines in Apache
Airflow
+This is the Airflow-specific how-to.
+For cron, systemd, Jenkins, a Kubernetes CronJob and the hop-run contract they
all share, see
xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling workflows
and pipelines].
+
== What is Apache Airflow?
From the https://airflow.apache.org/[Apache Airflow website]:
@@ -337,6 +340,8 @@ Your DAG logs will now show the environment variable and
the parameter we used i
So far, we've looked at DAG that we ran manually and ad-hoc. There are lots of
https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/index.html[well-documented^]
options to schedule DAGs in Apache Airflow. Since scheduling your DAGs is not
really Apache Hop related, we'll only cover this briefly here.
+For the same hop-run / short-lived-container contract with cron, systemd,
Jenkins or a Kubernetes CronJob, see
xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling workflows
and pipelines].
+
One option is to provide a cron string to schedule your DAG execution. For
example, to run a specific DAG at 10:00 am every morning, we'll change the
schedule_interval from None to a cron expression in the "with DAG" line in our
DAG (line breaks added for readability):
[source, python]
@@ -358,3 +363,6 @@ We've covered the basics of running Apache Hop pipelines
(or workflows) in Apach
There are other options: you could use Airflow's
https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html[BashOperator^]
to use xref:hop-run/index.adoc[hop-run] directly or the
https://airflow.apache.org/docs/apache-airflow-providers-http/stable/operators.html[HTTP
operator^] to run pipelines or workflows on a remote hop server.
+Airflow is one scheduler among several.
+xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines] covers cron, `at`, systemd, Windows Task Scheduler,
Jenkins and Kubernetes CronJob against the same hop-run contract.
+
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/scheduling-workflows-and-pipelines.adoc
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/scheduling-workflows-and-pipelines.adoc
new file mode 100644
index 0000000000..05e9477ea5
--- /dev/null
+++
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/scheduling-workflows-and-pipelines.adoc
@@ -0,0 +1,470 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[SchedulingWorkflowsAndPipelines]]
+:imagesdir: ../../assets/images
+:openvar: ${
+:closevar: }
+:description: This page shows how to start a workflow or pipeline from cron,
at, systemd, Windows Task Scheduler, Jenkins, a Kubernetes CronJob, or Apache
Airflow.
+
+= Scheduling workflows and pipelines
+
+Hop has no calendar of its own.
+xref:hop-server/index.adoc[Hop Server] does not queue work for later, and the
xref:workflow/actions/start.adoc[Start] action's repeat option is not a
scheduler.
+
+Something *outside* Hop decides *when* a pipeline or workflow runs: a crontab,
a systemd timer, Windows Task Scheduler, Jenkins, a Kubernetes CronJob, Apache
Airflow, or any other orchestrator you already use.
+That outside process starts xref:hop-run/index.adoc[hop-run] (or a short-lived
container that runs hop-run) and treats the exit code as the result.
+
+That is a feature, not a gap.
+Calendars, retries, overlapping-run policy, alerting and dependencies then
live in one place — the scheduler you already operate — instead of inside a
long-lived Java process.
+
+The xref:how-to-guides/run-hop-in-apache-airflow.adoc[Apache Airflow how-to]
is a full worked example of one of these options.
+This page is the catalogue: the command the scheduler should run, then a short
recipe for each common scheduler.
+
+== Choose how the run starts
+
+[cols="1,2,2",options="header"]
+|===
+|How the scheduler starts the work
+|When to use it
+|Hop side
+
+|xref:hop-run/index.adoc[hop-run] on a host
+|The host already has a Hop client, a registered project and an environment
+|The scheduler calls a small wrapper script.
+Exit code 0 is success.
+
+|A short-lived `apache/hop` (or project-baked) container
+|The project is already an image, or you do not want Hop installed on the
scheduler host
+|The scheduler runs `docker run --rm ...` or a Kubernetes Job.
+The container exits when hop-run exits.
+
+|Submit to a long-lived xref:hop-server/index.adoc[Hop Server]
+|Many clients share one always-on engine, or you need
xref:hop-server/web-service.adoc[web services]
+|The scheduler still calls hop-run, but with a
xref:pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc[remote]
run configuration.
+See xref:hop-server/deploying.adoc[Deploying Hop Server].
+|===
+
+For batch work, prefer the first or the second.
+A long-lived Hop Server is extra machinery if a scheduler already owns the
calendar — see
xref:hop-server/deploy-project-image.adoc#skip_hop_server_short_lived_containers[Skip
Hop Server: short-lived containers].
+
+[cols="1,1,3",options="header"]
+|===
+|Scheduler
+|Repeats
+|Typical fit
+
+|<<cron,cron>>
+|Yes
+|A Linux or macOS host that already has a Hop client
+
+|<<at,at>>
+|Once
+|A single future run on that same host (`at 22:00`, `at now + 2 hours`)
+
+|<<systemd,systemd timer>>
+|Yes
+|A Linux host that already uses systemd (the usual replacement for an ad-hoc
crontab)
+
+|<<windows,Windows Task Scheduler>>
+|Yes
+|A Windows host that already has a Hop client
+
+|<<jenkins,Jenkins>>
+|Yes
+|Jenkins already runs your CI; a scheduled job can run hop-run or `docker run`
+
+|<<cronjob,Kubernetes CronJob>>
+|Yes
+|The project is already a container image on a cluster
+
+|xref:how-to-guides/run-hop-in-apache-airflow.adoc[Apache Airflow]
+|Yes
+|You need DAG dependencies, sensors, retries or a shared data platform calendar
+|===
+
+GitHub Actions `on.schedule`, GitLab scheduled pipelines and similar CI timers
are the same idea as Jenkins: they start hop-run or a container on a calendar.
+They are CI systems first; use them when that is already where the work is
triggered.
+
+== Prerequisites
+
+For *hop-run on a host*:
+
+* A Hop client on that host (the same archive you unzip for Hop Gui), and Java
21 on `PATH` or in `HOP_JAVA_HOME` / `JAVA_HOME`.
+See xref:installation-configuration.adoc[Installation and configuration].
+* The project and a lifecycle environment registered in `HOP_CONFIG_FOLDER`,
so `-e prod` is enough.
+See xref:projects/projects-environments.adoc[Projects and environments] and
the xref:hop-server/deploy-git-checkout.adoc[git checkout] setup.
+* A run configuration that exists in the project's metadata (usually `local`).
+
+For a *short-lived container*:
+
+* An image that can run the project: the official `apache/hop` image with the
project mounted, or a xref:hop-server/deploy-project-image.adoc[project baked
into an image].
+* The xref:docker-container.adoc[short-lived environment variables]: at least
`HOP_FILE_PATH` and `HOP_RUN_CONFIG`.
+
+In both cases, send
xref:metadata-types/execution-information-location.adoc[execution information]
somewhere that outlives the process.
+A container (and often a cron job's working directory) is gone when the run
finishes.
+
+== A wrapper around hop-run
+
+Point the scheduler at a small script, not at `hop-run.sh` directly.
+The script is where you set `HOP_JAVA_HOME`, `HOP_CONFIG_FOLDER` and the log
file.
+
+That avoids three quoting traps:
+
+* *cron* treats `%` as a newline, so a hop-run command with date formats or
Windows-style `+%Y` does not belong in the crontab itself.
+* *systemd* expands `{openvar}NAME{closevar}` in `ExecStart=` from the unit
environment.
+A literal `{openvar}PROJECT_HOME{closevar}` for hop-run has to be escaped as
`${openvar}PROJECT_HOME{closevar}` (`$$` becomes `$`) — or you call a script
and never think about it.
+* The *shell* expands `{openvar}PROJECT_HOME{closevar}` if you double-quote it.
+hop-run resolves that variable *after* it enables the project, so the wrapper
must pass it in single quotes.
+
+ +
+
+[tabs]
+====
+Linux, macOS::
++
+--
+`/usr/local/bin/run-hop-nightly.sh`:
+
+[source,bash]
+----
+#!/usr/bin/env bash
+set -euo pipefail
+
+export HOP_JAVA_HOME=/usr/lib/jvm/java-21-openjdk
+export HOP_CONFIG_FOLDER=/etc/hop/config
+
+HOP=/opt/hop
+mkdir -p /var/log/hop
+LOG=/var/log/hop/nightly-$(date +%Y%m%d-%H%M%S).log
+
+"${HOP}/hop-run.sh" \
+ -e prod \
+ -r local \
+ -f '{openvar}PROJECT_HOME{closevar}/main.hwf' \
+ -l Basic \
+ -lf "${LOG}" \
+ >/dev/null 2>&1
+----
+
+`chmod +x /usr/local/bin/run-hop-nightly.sh` and run it once by hand before
you schedule it.
+`--logfile` (`-lf`) *overwrites* the file, which is why the name includes a
timestamp.
+It adds a file listener *alongside* the console logger, so hop-run still
writes to stdout unless you redirect it as above.
+cron mails on any output, not on a non-zero exit, and without that redirect a
successful run would mail the whole log.
+
+On Linux, put `flock -n /tmp/hop-nightly.lock` in front of the script so a
second start fails instead of running the same load twice (`flock` is not on
macOS by default).
+Otherwise use the scheduler's own "do not start a second instance" option,
shown in each recipe below.
+--
+
+Windows::
++
+--
+`C:\hop-jobs\run-hop-nightly.cmd`:
+
+[source,bat]
+----
+@echo off
+setlocal
+
+set "HOP_JAVA_HOME=C:\Program Files\Microsoft\jdk-21"
+set "HOP_CONFIG_FOLDER=C:\hop-config"
+set "HOP=C:\hop"
+
+cd /d "%HOP%"
+call hop-run.bat -e prod -r local -f {openvar}PROJECT_HOME{closevar}/main.hwf
-l Basic
+exit /b %ERRORLEVEL%
+----
+
+`call` is required: without it, control never returns from `hop-run.bat` and
any lines after it (cleanup, `exit /b`) are skipped.
+
+Task Scheduler's setting *If the task is already running: Do not start a new
instance* is the overlap policy on Windows.
+--
+====
+
+Use `-e` (the lifecycle environment) rather than `-j` (the project).
+The environment already points at the project, and it is what differs between
Development, Test and Production.
+
+xref:hop-run/index.adoc#_possible_exit_codes[hop-run exit codes]: `0` success,
`1` the workflow or pipeline failed, `2` hop-run itself failed, `9` bad
arguments.
+Every scheduler below treats a non-zero exit as a failed run.
+
+== Recipes
+
+[#cron]
+=== cron
+
+[source]
+----
+# 02:00 every day, host local time.
+# Linux: flock refuses a second start while the previous run still holds the
lock.
+# cron mails on any stdout/stderr, not on exit code; the wrapper keeps those
quiet.
+0 2 * * * /usr/bin/flock -n /tmp/hop-nightly.lock
/usr/local/bin/run-hop-nightly.sh
+----
+
+Install with `crontab -e` as the user that should run Hop (not root, unless
that user owns the project files).
+cron's environment is minimal: `PATH` is often just `/usr/bin:/bin`, and it
does not load `~/.bashrc`.
+That is why `HOP_JAVA_HOME` and `HOP_CONFIG_FOLDER` live in the wrapper, not
in the crontab.
+
+cron uses the host's local timezone.
+Put the real command in the script; keep `%` out of the crontab line.
+
+[#at]
+=== at
+
+`at` runs a command *once* at a future time.
+It is the one-shot counterpart of cron, and it is what you want for "run this
load tonight at 22:00" or "run it two hours from now".
+The package is often not installed by default (`apt install at` / `dnf install
at`).
+
+[source,bash]
+----
+echo /usr/local/bin/run-hop-nightly.sh | at 22:00
+echo /usr/local/bin/run-hop-nightly.sh | at now + 2 hours
+echo /usr/local/bin/run-hop-nightly.sh | at 02:00 tomorrow
+----
+
+`atq` lists pending jobs, `atrm <job>` cancels one.
+The job runs with the environment `at` captured when you submitted it, which
is closer to your interactive shell than cron is — still prefer the wrapper so
Java and `HOP_CONFIG_FOLDER` are explicit.
+
+[#systemd]
+=== systemd timer
+
+A `Type=oneshot` service plus a timer is the systemd equivalent of a crontab
line.
+systemd will not start a second instance of the same service while the first
is still running.
+
+`/etc/systemd/system/hop-nightly.service`:
+
+[source,ini]
+----
+[Unit]
+Description=Nightly Hop load
+After=network.target
+
+[Service]
+Type=oneshot
+User=hop
+Group=hop
+ExecStart=/usr/local/bin/run-hop-nightly.sh
+----
+
+`/etc/systemd/system/hop-nightly.timer`:
+
+[source,ini]
+----
+[Unit]
+Description=Run the nightly Hop load at 02:00
+
+[Timer]
+OnCalendar=*-*-* 02:00:00
+Persistent=true
+RandomizedDelaySec=90
+
+[Install]
+WantedBy=timers.target
+----
+
+`Persistent=true` runs a missed job after a reboot.
+`RandomizedDelaySec` spreads the start if many timers fire at midnight.
+
+[source,bash]
+----
+sudo systemctl daemon-reload
+sudo systemctl enable --now hop-nightly.timer
+systemctl list-timers hop-nightly.timer
+journalctl -u hop-nightly.service
+----
+
+A long-lived Hop Server is a different unit (`Type=simple`,
`Restart=on-failure`).
+A sample is in the repository at
`docs/hop-user-manual/modules/ROOT/assets/files/hop-server/hop-server.service`.
+
+[#windows]
+=== Windows Task Scheduler
+
+. Open *Task Scheduler* and choose *Create Task* (not *Create Basic Task* —
you need the overlap setting).
+. *General*: name it, choose *Run whether user is logged on or not*, and run
as the account that can read the project and `HOP_CONFIG_FOLDER`.
+. *Triggers*: *New* → *Daily* at 02:00, or *One time* for the equivalent of
`at`.
+. *Actions*: *Start a program* → `C:\hop-jobs\run-hop-nightly.cmd`.
+Set *Start in* to `C:\hop` so relative paths inside hop-run.bat resolve.
+. *Settings*: *If the task is already running, then the following rule
applies: Do not start a new instance*.
+
+From a command prompt the same daily task is:
+
+[source,bat]
+----
+schtasks /create /tn "Hop nightly" /sc daily /st 02:00 /ru hop /rp * /tr
"C:\hop-jobs\run-hop-nightly.cmd"
+----
+
+`schtasks /create /sc once /st 22:00 ...` is the one-shot form.
+`schtasks /run /tn "Hop nightly"` runs it immediately; `schtasks /query /tn
"Hop nightly" /v` shows the last result.
+
+[#jenkins]
+=== Jenkins
+
+Jenkins as a *scheduler* is a job with a cron trigger that runs hop-run or
`docker run`.
+That is a different use of Jenkins from
xref:hop-server/deploy-project-image.adoc[building a project image], which is
CI.
+
+A *Freestyle project* is enough: *Build Triggers* → *Build periodically* with
a cron string (`0 2 * * *`), then an *Execute shell* (or *Execute Windows batch
command*) step that calls the wrapper.
+Check *Do not allow concurrent builds* so two ticks cannot overlap.
+
+The same job as a Pipeline:
+
+[source,groovy]
+----
+pipeline {
+ agent any
+ triggers { cron('0 2 * * *') }
+ options { disableConcurrentBuilds() }
+ stages {
+ stage('Hop') {
+ steps {
+ sh '/usr/local/bin/run-hop-nightly.sh'
+ }
+ }
+ }
+}
+----
+
+If the agent should not have a Hop client, start a short-lived container
instead (see <<container>> below) from that `sh` step.
+Pin the image tag; do not use `latest` on a calendar.
+
+[#container]
+=== Short-lived container
+
+The official image runs hop-run and exits when `HOP_FILE_PATH` and
`HOP_RUN_CONFIG` are set.
+Full variable list: xref:docker-container.adoc[Hop in Docker].
+
+[source,bash]
+----
+docker run --rm \
+ -e HOP_LOG_LEVEL=Basic \
+ -e HOP_PROJECT_FOLDER=/files \
+ -e HOP_PROJECT_NAME=your-project \
+ -e HOP_ENVIRONMENT_NAME=prod \
+ -e HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS=/config/prod.json \
+ -e HOP_FILE_PATH='{openvar}PROJECT_HOME{closevar}/main.hwf' \
+ -e HOP_RUN_CONFIG=local \
+ -v /opt/hop-projects/your-project:/files:ro \
+ -v /etc/hop/environments/prod.json:/config/prod.json:ro \
+ apache/hop:<tag>
+----
+
+Replace `<tag>` with a release tag (not `latest`).
+Replace the two bind mounts with nothing extra when CI already
xref:hop-server/deploy-project-image.adoc[baked the project into the image];
then `HOP_PROJECT_FOLDER` is a path inside the image such as `/your-project`.
+
+cron, systemd, Jenkins and Airflow can all run that `docker run --rm` line.
+On a cluster the same contract is a Job.
+
+[#cronjob]
+=== Kubernetes CronJob
+
+[source,yaml]
+----
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+ name: hop-nightly
+spec:
+ schedule: "0 2 * * *"
+ timeZone: "Europe/Brussels"
+ concurrencyPolicy: Forbid
+ successfulJobsHistoryLimit: 3
+ failedJobsHistoryLimit: 3
+ jobTemplate:
+ spec:
+ backoffLimit: 1
+ template:
+ spec:
+ restartPolicy: Never
+ containers:
+ - name: hop
+ image: registry.example.com/your-project-hop:1.2.3
+ env:
+ - name: HOP_LOG_LEVEL
+ value: Basic
+ - name: HOP_PROJECT_FOLDER
+ value: /your-project
+ - name: HOP_PROJECT_NAME
+ value: your-project
+ - name: HOP_ENVIRONMENT_NAME
+ value: prod
+ - name: HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS
+ value: /config/prod.json
+ - name: HOP_FILE_PATH
+ value: '{openvar}PROJECT_HOME{closevar}/main.hwf'
+ - name: HOP_RUN_CONFIG
+ value: local
+ volumeMounts:
+ - name: env
+ mountPath: /config
+ readOnly: true
+ volumes:
+ - name: env
+ secret:
+ secretName: hop-prod-env
+----
+
+`concurrencyPolicy: Forbid` is the overlap lock.
+`backoffLimit: 1` retries once immediately; further retries belong to the next
calendar tick (or to Airflow).
+`timeZone` needs a current enough Kubernetes; without it the schedule is UTC.
+The xref:hop-server/deploy-project-image.adoc[project-in-an-image] page is the
image this CronJob runs.
+
+=== Apache Airflow
+
+Airflow is the right scheduler when the Hop run is one task among
dependencies, sensors and retries.
+The xref:how-to-guides/run-hop-in-apache-airflow.adoc[Airflow how-to] uses the
DockerOperator against a short-lived `apache/hop` container — the same contract
as <<container>>.
+
+Two other honest options, covered there and on
xref:hop-server/deploy-export-resources.adoc[Remote run with export resources]:
+
+* `BashOperator` calling hop-run on a worker that has a Hop client (the
wrapper in this page).
+* hop-run with a *remote* run configuration, so Airflow submits work to a Hop
Server instead of executing it on the worker.
+
+A cron expression on the DAG (`schedule='0 2 * * *'`, or `schedule_interval`
on older Airflow) is Airflow's equivalent of the crontab line above.
+Prefer Airflow's own DAG schedule over also wrapping the DAG in cron.
+
+== What not to use as a scheduler
+
+The xref:workflow/actions/start.adoc[Start] action can repeat a workflow on an
interval or at a clock time.
+That option exists for historical reasons.
+It keeps the Java process running for as long as the workflow is "scheduled",
holds memory, and has no overlap policy, no durable calendar and no alerting.
+
+Use it only as a last resort on a workstation.
+In production, start the workflow once from hop-run and let the scheduler
start it again next time.
+
+== Things every scheduler has to get right
+
+* *The process environment.*
+cron in particular will not see the `JAVA_HOME` or `PATH` from your login
shell.
+Set `HOP_JAVA_HOME` (or `JAVA_HOME`) and `HOP_CONFIG_FOLDER` in the wrapper or
the unit, not by hoping they are inherited.
+* *Overlapping runs.*
+A load that still runs at 02:00 the next day will corrupt results if a second
copy starts.
+`flock`, systemd's default, Task Scheduler's *Do not start a new instance*,
Jenkins `disableConcurrentBuilds()`, Kubernetes `concurrencyPolicy: Forbid`,
Airflow `max_active_runs=1` — pick one and make it explicit.
+* *Logs that outlive the process.*
+hop-run writes to stdout (which systemd, Jenkins and Kubernetes already
collect) and optionally to `-lf`.
+`-lf` does not stop the console logger: cron mails on any leftover stdout, not
on a non-zero exit, so the wrapper above discards it.
+Configure an xref:metadata-types/execution-information-location.adoc[execution
information location] on the run configuration so you can inspect the run later
from the xref:hop-gui/perspective-execution-information.adoc[Execution
Information perspective].
+* *Timezones.*
+cron and systemd use the host timezone; Kubernetes CronJob defaults to UTC;
Airflow uses the DAG timezone.
+Write the intended zone down next to the expression.
+* *Image tags and Hop versions.*
+Pin them.
+`latest` on a calendar turns an unrelated pull into an unplanned production
upgrade.
+
+== See also
+
+* xref:hop-run/index.adoc[hop-run] — the command, its options and exit codes
+* xref:docker-container.adoc[Hop in Docker] — short-lived vs long-lived
containers
+* xref:how-to-guides/run-hop-in-apache-airflow.adoc[Run Hop workflows and
pipelines in Apache Airflow]
+* xref:hop-server/deploying.adoc[Deploying Hop Server] — when you *do* want a
long-lived server, and when you do not
+* xref:cloud.adoc[Running Apache Hop in the cloud] — the same "what starts a
run" decision on a cloud platform
+* xref:workflow/actions/start.adoc[Start action] — why its repeat option is
not a scheduler
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-unit-testing.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-unit-testing.adoc
index fdeae55e61..79d7b5290e 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-unit-testing.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-unit-testing.adoc
@@ -238,7 +238,7 @@ image::pipeline-unit-testing-bypass-transform.png[Unit Test
- Bypass transform,w
There is a workflow action called "Run pipeline unit tests" which can execute
all defined unit tests of a certain type.
The output of the transform can be stored in any format or location with
regular Hop transforms.
-Execute the workflow through hop-run, in a scheduler or through a CI/CD
pipeline in e.g. Jenkins.
+Execute the workflow through hop-run, in a
xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[scheduler] or
through a CI/CD pipeline in e.g. Jenkins.
Use the 'Get test names' in this action to specify which of the available unit
tests you want to include in your workflow.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/governance.adoc
b/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/governance.adoc
index 9c88799bf6..6e38aed031 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/governance.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/governance.adoc
@@ -26,4 +26,5 @@ The items below will make your Apache Hop project easier to
manage, to monitor a
* Run continuous integration
* Set up lifecycle environments (development, test, acceptance, production)
* Deploy Hop Server using one of the xref:hop-server/deploying.adoc[documented
patterns]
+* Schedule production runs with an
xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[external scheduler],
not the Start action's repeat option
* Test your pipelines with xref:pipeline/pipeline-unit-testing.adoc[unit
tests]. Run all your unit tests regularly, validate the results & take action
if needed
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/fileexists.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/fileexists.adoc
index 5c1ffa8029..c72a9afdd0 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/fileexists.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/fileexists.adoc
@@ -36,7 +36,7 @@ The first part runs in PERL.
You have batch scripts that accesses data from a remote location, performs
first-level row processing, and outputs the data to a specified directory.
-You do not want to start the workflow until this is done, so you put the
workflow on a scheduler.
+You do not want to start the workflow until this is done, so you put the
workflow on a
xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[scheduler].
As soon as the task is complete, the file is placed in a well-known location
so that the "file exists." That is the signal that launches the workflow for
final processing.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/start.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/start.adoc
index de71432773..ca03439c54 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/start.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/start.adoc
@@ -30,7 +30,7 @@ NOTE: Every workflow needs to have one and only one start
action.
image::actions/action-start-dialog.png[The alt text of the image]
-WARNING: The start action has additional options to repeat a workflow. This
works in both the local and remote workflow engine, but is mainly available for
historical reasons and shouldn't be considered as an alternative to `cron`,
Apache Airflow or any other type of scheduling.
+WARNING: The start action has additional options to repeat a workflow. This
works in both the local and remote workflow engine, but is mainly available for
historical reasons and shouldn't be considered as an alternative to `cron`,
Apache Airflow or any other type of scheduling. See
xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling workflows
and pipelines].
When you activate the repeat option the workflow will continue to run and
restart based on the interval you specify. This can be on a fixed day with a
specific timestamp. Or every x seconds/minutes
