This is an automated email from the ASF dual-hosted git repository.
kaxil pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new b503e692da0 Fix `SSHRemoteJobOperator` on Windows hosts (#73352)
b503e692da0 is described below
commit b503e692da0d94fdca0ebd177b63c8de6deeb432
Author: Timo <[email protected]>
AuthorDate: Wed Nov 4 03:39:52 2026 -0800
Fix `SSHRemoteJobOperator` on Windows hosts (#73352)
* fix for windows
* move comment out of the block that gets sent to remote host
* Fix command quoting, log streaming and literal paths on Windows
Follow-up fixes on top of the Windows wrapper rewrite, each checked on
windows-2022 and windows-latest runners (Windows PowerShell 5.1):
- Inline the command into the job script as written. The script travels
in a verbatim here-string, so backtick escaping reached PowerShell
literally and broke quoted arguments.
- Run the command inside & { } so every statement's output is logged,
not only the last one's.
- Write the log through a shared, auto-flushed UTF-8 writer. Add-Content
kept the file locked until the command ended, so the trigger could not
tail it while the job ran.
- Fail the submission when Win32_Process.Create cannot start the job,
and start it in the SSH session's working directory, not System32.
- Pass job paths to PowerShell literally: single-quote everything except
a leading $env:NAME, double PowerShell's typographic quotes, and use
-LiteralPath so brackets are not read as wildcards. The same quoting
now applies to environment values.
* Run the Windows command as its own script and fail jobs instead of hanging
The command is written to cmd.ps1 and called from the job script inside
try/catch/finally, so the exit code is recorded however the command ends:
an inline `exit N` keeps N (skip_on_exit_code depends on it), a throw or
a parse error fails the job with the message in the log, and a crashed
process's negative code is recorded unsigned so the trigger accepts it.
A here-string in the command no longer ends the wrapper's own.
The job script is started with -Command instead of -File, so a Group
Policy that only allows signed scripts cannot stop the job's bookkeeping;
under such a policy the job fails with the policy error in its log.
*>&1 keeps Write-Host and warnings, and objects are formatted the way a
console shows them. The wrapper stops on the first setup error and
reports it on stderr, so a bad remote_base_dir fails the submission
instead of reporting a job that never starts. A leading $Env:NAME is
expanded in any casing.
Checked on windows-2022 and windows-latest runners, including a machine
policy of AllSigned.
* Keep Windows job logs readable through SSH and owned by the job
The log tail now writes the chunk to stdout as UTF-8 bytes. Printed as a
PowerShell string, it was re-encoded in the SSH session's console code
page (437 on a default install), which the trigger's strict UTF-8
decoding rejects, so any non-ASCII log line put the trigger into a
reconnect loop until the task failed.
The job script opens its log through a local path, so an environment
entry named LOG_FILE can no longer redirect it, and the launch command
quotes the already-resolved job directory instead of re-expanding
$env:NAME in the new process's environment.
Document the Windows exit code rules, the Group Policy requirement and
the inline command length that fits through cmd.exe.
* Test the runtime launch quoting and the byte-level log tail
Emulate the wrapper's runtime quoting of the job directory and read the
launch command back, pin the log tail's byte write, and document what
shares the cmd.exe command line budget on Windows.
---------
Co-authored-by: Kaxil Naik <[email protected]>
---
providers/ssh/docs/operators/ssh_remote_job.rst | 23 ++-
.../providers/ssh/operators/ssh_remote_job.py | 6 +-
.../src/airflow/providers/ssh/utils/remote_job.py | 174 ++++++++++------
.../ssh/tests/unit/ssh/utils/test_remote_job.py | 227 ++++++++++++++++++++-
4 files changed, 365 insertions(+), 65 deletions(-)
diff --git a/providers/ssh/docs/operators/ssh_remote_job.rst
b/providers/ssh/docs/operators/ssh_remote_job.rst
index f675bf6288e..a21106fac2a 100644
--- a/providers/ssh/docs/operators/ssh_remote_job.rst
+++ b/providers/ssh/docs/operators/ssh_remote_job.rst
@@ -62,7 +62,7 @@ How It Works
1. **Job Submission**: The operator connects via SSH and submits a wrapper
script that:
* Creates a unique job directory on the remote host
- * Starts your command as a detached process (``nohup`` on POSIX,
``Start-Process`` on Windows)
+ * Starts your command as a detached process (``nohup`` on POSIX,
``Win32_Process.Create`` on Windows)
* Redirects output to a log file
* Writes exit code to a file when complete
@@ -157,6 +157,9 @@ Parameters
* POSIX: ``/tmp/airflow-ssh-jobs``
* Windows: ``$env:TEMP\\airflow-ssh-jobs``
+ On Windows a leading ``$env:NAME`` is expanded on the remote host; the rest
of the path is
+ used literally, so ``$``, quotes and brackets later in the path are part of
the directory name.
+
* ``poll_interval`` (int, optional): How often to check job status in seconds.
Default: 5
* ``log_chunk_size`` (int, optional): Maximum bytes to read from log per poll.
Default: 65536
* ``timeout`` (int, optional): Hard timeout for the entire task in seconds.
Default: None (no timeout)
@@ -190,6 +193,20 @@ For Windows, ensure:
* OpenSSH Server is installed and running
* PowerShell is available (default on modern Windows)
* SSH connection allows command execution
+* No Group Policy restricts PowerShell to signed scripts or turns scripts off.
Your command runs as
+ a script with ``-ExecutionPolicy Bypass``, which a Group Policy setting
overrides; under such a
+ policy the job fails with the policy error in its log.
+
+On Windows the exit code is the one passed to ``exit``, or else that of the
last native program the
+command ran, or 0. A cmdlet error that does not stop the command does not fail
the task; set
+``$ErrorActionPreference = 'Stop'`` or ``exit 1`` when it should. A negative
exit code, which is how
+Windows reports a crashed process, is recorded as its unsigned value (``-1``
becomes ``4294967295``),
+so use that value in ``skip_on_exit_code``.
+
+The command travels through ``cmd.exe``, the default OpenSSH shell, whose
command line limit leaves
+room for an inline command of roughly 1,000 characters. Environment values, a
long base directory and
+long Dag, task and run ids use the same budget, and a single quote counts
twice. For anything longer,
+put it in a script on the host and call that.
Job Directory Structure
-----------------------
@@ -204,6 +221,8 @@ Each job creates a directory on the remote host with these
files:
├── pid # Process ID (for on_kill)
└── status # Optional status file (for user scripts)
+On Windows the directory also holds ``cmd.ps1``, your command, and
``job.ps1``, which runs it.
+
Your command can access these via environment variables:
* ``LOG_FILE``: Path to the log file
@@ -228,7 +247,7 @@ the initial job submission must succeed. The connection
used for submission is r
(``conn_retry_attempts``); if every attempt fails, the task fails immediately.
The trigger also
reconnects automatically if the monitoring connection drops mid-job.
-**Remote Process Management**: Jobs are detached using ``nohup`` (POSIX) or
``Start-Process`` (Windows).
+**Remote Process Management**: Jobs are detached using ``nohup`` (POSIX) or
``Win32_Process.Create`` (Windows).
If the remote host reboots during job execution, the job will be lost.
**Log Size**: Large log outputs may impact performance. The ``log_chunk_size``
parameter controls
diff --git
a/providers/ssh/src/airflow/providers/ssh/operators/ssh_remote_job.py
b/providers/ssh/src/airflow/providers/ssh/operators/ssh_remote_job.py
index eab60940600..54eb2445219 100644
--- a/providers/ssh/src/airflow/providers/ssh/operators/ssh_remote_job.py
+++ b/providers/ssh/src/airflow/providers/ssh/operators/ssh_remote_job.py
@@ -56,7 +56,7 @@ class SSHRemoteJobOperator(BaseOperator):
continues running independently of the SSH connection.
The remote job is wrapped to:
- - Run detached from the SSH session (via nohup on POSIX, Start-Process on
Windows)
+ - Run detached from the SSH session (via nohup on POSIX, Win32_Process on
Windows)
- Redirect stdout/stderr to a log file
- Write the exit code to a file on completion
@@ -67,7 +67,9 @@ class SSHRemoteJobOperator(BaseOperator):
Values are passed as literal strings; the remote shell does not expand
``$VAR``
or ``$(...)`` in them.
:param remote_base_dir: Base directory for job artifacts (templated).
- Defaults to /tmp/airflow-ssh-jobs on POSIX,
C:\\Windows\\Temp\\airflow-ssh-jobs on Windows
+ Defaults to /tmp/airflow-ssh-jobs on POSIX,
$env:TEMP\\airflow-ssh-jobs on Windows.
+ On Windows a leading ``$env:NAME`` is expanded on the remote host and
the rest of
+ the path is used literally.
:param poll_interval: Seconds between status polls (default: 5)
:param log_chunk_size: Max bytes to read per poll (default: 65536)
:param timeout: Hard timeout in seconds for the entire operation
diff --git a/providers/ssh/src/airflow/providers/ssh/utils/remote_job.py
b/providers/ssh/src/airflow/providers/ssh/utils/remote_job.py
index 557bfef679e..cea3ad5b521 100644
--- a/providers/ssh/src/airflow/providers/ssh/utils/remote_job.py
+++ b/providers/ssh/src/airflow/providers/ssh/utils/remote_job.py
@@ -30,6 +30,30 @@ from typing import Literal
POSIX_DEFAULT_BASE_DIR = "/tmp/airflow-ssh-jobs"
WINDOWS_DEFAULT_BASE_DIR = "$env:TEMP\\airflow-ssh-jobs"
+# PowerShell ends a single-quoted string at any of these, not only the ASCII
+# apostrophe; a doubled one stands for itself.
+_PS_SINGLE_QUOTES = re.compile("(['\u2018\u2019\u201a\u201b])")
+_PS_ENV_PREFIX = re.compile(r"\$env:([A-Za-z_][A-Za-z0-9_]*)", re.IGNORECASE)
+
+
+def _ps_quote(value: str) -> str:
+ """Quote ``value`` as a PowerShell single-quoted string, which is never
interpolated."""
+ return "'" + _PS_SINGLE_QUOTES.sub(r"\1\1", value) + "'"
+
+
+def _ps_path(path: str) -> str:
+ """
+ Return a PowerShell expression that evaluates to ``path``.
+
+ A leading ``$env:NAME``, as in the default base directory, is expanded on
the remote
+ host. The rest of the path is literal, so a ``$``, backtick or quote in
+ ``remote_base_dir`` stays part of the directory name. Pass the result to
+ ``-LiteralPath``, since ``-Path`` would read ``[`` and ``]`` as wildcards.
+ """
+ if match := _PS_ENV_PREFIX.match(path):
+ return f"($env:{match.group(1)} + {_ps_quote(path[match.end() :])})"
+ return _ps_quote(path)
+
def _validate_job_dir(
job_dir: str, remote_os: Literal["posix", "windows"], base_dir: str | None
= None
@@ -249,15 +273,23 @@ def build_windows_wrapper_command(
environment: dict[str, str] | None = None,
) -> str:
"""
- Build a PowerShell wrapper that runs the command detached via
Start-Process.
+ Build a PowerShell wrapper that runs the command detached via
``Win32_Process.Create``.
The wrapper:
- - Creates the job directory
- - Starts the command in a new detached PowerShell process
- - Redirects stdout/stderr to the log file
- - Writes the exit code atomically on completion
- - Writes the PID for potential cancellation
+ - Creates the job directory and writes the command into it as ``cmd.ps1``,
next to
+ the job script that runs it
+ - Starts the job script as a separate process that outlives the SSH
session, in the
+ SSH session's working directory
+ - Appends everything the command writes, on every stream, to the log file
as UTF-8
+ - Writes the exit code on completion, including when the command calls
``exit``,
+ throws, or fails to parse
+ - Writes the PID for potential cancellation, and fails the submission if
the job
+ directory cannot be set up or the process cannot be started
+
+ The command runs as a script, so the host's execution policy applies to it
with
+ ``-ExecutionPolicy Bypass`` for the process; a Group Policy that only
allows signed
+ scripts fails the job with the policy error in its log.
:param command: The command to execute (PowerShell script path or command)
:param paths: RemoteJobPaths instance with all paths
@@ -268,42 +300,68 @@ def build_windows_wrapper_command(
if environment:
for key, value in environment.items():
_validate_env_var_name(key)
- escaped_value = value.replace("'", "''")
- env_setup += f"$env:{key} = '{escaped_value}'; "
-
- def ps_escape(s: str) -> str:
- return s.replace("'", "''")
-
- job_dir = ps_escape(paths.job_dir)
- log_file = ps_escape(paths.log_file)
- exit_code_file = ps_escape(paths.exit_code_file)
- exit_code_tmp = ps_escape(paths.exit_code_tmp_file)
- pid_file = ps_escape(paths.pid_file)
- status_file = ps_escape(paths.status_file)
- escaped_command = ps_escape(command)
- job_id = ps_escape(paths.job_id)
-
- child_script = f"""$ErrorActionPreference = 'Continue'
-$env:LOG_FILE = '{log_file}'
-$env:STATUS_FILE = '{status_file}'
+ env_setup += f"$env:{key} = {_ps_quote(value)}; "
+
+ sep = paths.sep
+
+ # Use $d and $b for directories to avoid passing long paths multiple
times, and write
+ # the scripts to disk to avoid double base64 encoding. job.ps1 travels in
a verbatim
+ # here-string and holds nothing user-supplied but quoted environment
values; the command
+ # goes to its own cmd.ps1, so `exit N` in it ends only that script, with
$LASTEXITCODE N.
+ # *>&1 keeps Write-Host and warnings; error records are turned into their
message before
+ # Out-String formats objects the way a console would. The log writer
shares the file for
+ # reading and flushes every line, so the trigger can tail it while the job
runs.
+ # Windows reports a crashed process as a negative exit code; record it
unsigned, since
+ # the trigger only accepts digits. The writer opens $logPath, not
$env:LOG_FILE, so an
+ # environment entry of that name cannot redirect the job's log.
+ child_script = f"""$ErrorActionPreference = "Continue"
+$logPath = "$b{sep}stdout.log"
+$env:LOG_FILE = $logPath
+$env:STATUS_FILE = "$b{sep}status"
{env_setup}
-{escaped_command}
-$ec = $LASTEXITCODE
-if ($null -eq $ec) {{ $ec = 0 }}
-Set-Content -NoNewline -Path '{exit_code_tmp}' -Value $ec
-Move-Item -Force -Path '{exit_code_tmp}' -Destination '{exit_code_file}'
+$log = [System.IO.StreamWriter]::new([System.IO.FileStream]::new($logPath,
'Append', 'Write', 'ReadWrite'), [System.Text.UTF8Encoding]::new($false))
+$log.AutoFlush = $true
+$ec = 1
+try {{
+ & "$b{sep}cmd.ps1" *>&1 |
+ ForEach-Object {{ if ($_ -is [System.Management.Automation.ErrorRecord])
{{ "$_" }} else {{ $_ }} }} |
+ Out-String -Stream |
+ ForEach-Object {{ $log.WriteLine($_) }}
+ $ec = if ($null -eq $LASTEXITCODE) {{ 0 }} else {{ $LASTEXITCODE }}
+}} catch {{
+ $log.WriteLine("$_")
+}} finally {{
+ $log.Close()
+ if ($ec -lt 0) {{ $ec += 4294967296 }}
+ Set-Content -NoNewline -LiteralPath "$b{sep}exit_code.tmp" -Value $ec
+ Move-Item -Force -LiteralPath "$b{sep}exit_code.tmp" -Destination
"$b{sep}exit_code"
+}}
"""
- child_script_bytes = child_script.encode("utf-16-le")
- encoded_script = base64.b64encode(child_script_bytes).decode("ascii")
-
- wrapper = f"""$jobDir = '{job_dir}'
-New-Item -ItemType Directory -Force -Path $jobDir | Out-Null
-$log = '{log_file}'
-'' | Set-Content -Path $log
-$p = Start-Process -FilePath 'powershell.exe' -ArgumentList @('-NoProfile',
'-NonInteractive', '-EncodedCommand', '{encoded_script}')
-RedirectStandardOutput $log -RedirectStandardError $log -PassThru -WindowStyle
Hidden
-Set-Content -NoNewline -Path '{pid_file}' -Value $p.Id
-Write-Output '{job_id}'
+ # Launch as Win32_Process so that it doesn't die when SSH exits. job.ps1
is read and run
+ # as a script block rather than with -File, so execution policy never
blocks the job's
+ # own bookkeeping, only the user's cmd.ps1. The launch quotes the
already-resolved $d, so
+ # the child does not re-expand $env:NAME in its own environment. Create
reports failure
+ # in ReturnValue instead of throwing.
+ wrapper = f"""$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+trap {{ [Console]::Error.WriteLine("$_"); exit 1 }}
+$d = {_ps_path(paths.job_dir)}
+New-Item -ItemType Directory -Force -Path $d | Out-Null
+[System.IO.File]::WriteAllText("$d{sep}cmd.ps1", {_ps_quote(command)},
[System.Text.UTF8Encoding]::new($true))
+@'
+{child_script}'@ | Set-Content -LiteralPath "$d{sep}job.ps1" -Encoding UTF8
+New-Item -ItemType File -Force -Path "$d{sep}stdout.log" | Out-Null
+
+$q = "'" + ($d -replace "{_PS_SINGLE_QUOTES.pattern}", '$1$1') + "'"
+$launch = 'powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass
-Command "$b = ' + $q + '; . ([scriptblock]::Create([IO.File]::ReadAllText($b +
''{sep}job.ps1'')))"'
+$p = ([wmiclass]'Win32_Process').Create($launch, $PWD.ProviderPath, $null)
+if ($p.ReturnValue -ne 0) {{
+ [Console]::Error.WriteLine("Win32_Process.Create failed with return value
$($p.ReturnValue)")
+ exit 1
+}}
+Set-Content -NoNewline -LiteralPath "$d{sep}pid" -Value $p.ProcessId
+Write-Output {_ps_quote(paths.job_id)}
"""
wrapper_bytes = wrapper.encode("utf-16-le")
encoded_wrapper = base64.b64encode(wrapper_bytes).decode("ascii")
@@ -329,14 +387,17 @@ def build_windows_log_tail_command(log_file: str, offset:
int, max_bytes: int) -
"""
Build a PowerShell command to read log bytes from offset.
+ The chunk is written to stdout as UTF-8 bytes. Printing it as a PowerShell
string would
+ re-encode it in the SSH session's console code page, which the trigger's
strict UTF-8
+ decoding rejects for any non-ASCII character.
+
:param log_file: Path to the log file
:param offset: Byte offset to start reading from
:param max_bytes: Maximum bytes to read
:return: PowerShell command that outputs the log chunk
"""
- escaped_path = log_file.replace("'", "''")
- script = f"""$path = '{escaped_path}'
-if (Test-Path $path) {{
+ script = f"""$path = {_ps_path(log_file)}
+if (Test-Path -LiteralPath $path) {{
try {{
$fs = [System.IO.File]::Open($path, 'Open', 'Read', 'ReadWrite')
$fs.Seek({offset}, [System.IO.SeekOrigin]::Begin) | Out-Null
@@ -344,7 +405,10 @@ if (Test-Path $path) {{
$n = $fs.Read($buf, 0, $buf.Length)
$fs.Close()
if ($n -gt 0) {{
- [System.Text.Encoding]::UTF8.GetString($buf, 0, $n)
+ $bytes =
[System.Text.Encoding]::UTF8.GetBytes([System.Text.Encoding]::UTF8.GetString($buf,
0, $n))
+ $out = [Console]::OpenStandardOutput()
+ $out.Write($bytes, 0, $bytes.Length)
+ $out.Flush()
}}
}} catch {{}}
}}"""
@@ -371,10 +435,9 @@ def build_windows_file_size_command(file_path: str) -> str:
:param file_path: Path to the file
:return: PowerShell command that outputs the file size
"""
- escaped_path = file_path.replace("'", "''")
- script = f"""$path = '{escaped_path}'
-if (Test-Path $path) {{
- (Get-Item $path).Length
+ script = f"""$path = {_ps_path(file_path)}
+if (Test-Path -LiteralPath $path) {{
+ (Get-Item -LiteralPath $path).Length
}} else {{
0
}}"""
@@ -401,10 +464,9 @@ def build_windows_completion_check_command(exit_code_file:
str) -> str:
:param exit_code_file: Path to the exit code file
:return: PowerShell command that outputs exit code if done, empty otherwise
"""
- escaped_path = exit_code_file.replace("'", "''")
- script = f"""$path = '{escaped_path}'
-if (Test-Path $path) {{
- $txt = Get-Content -Raw -Path $path
+ script = f"""$path = {_ps_path(exit_code_file)}
+if (Test-Path -LiteralPath $path) {{
+ $txt = Get-Content -Raw -LiteralPath $path
if ($txt -match '^[0-9]+$') {{ $txt.Trim() }}
}}"""
script_bytes = script.encode("utf-16-le")
@@ -454,10 +516,9 @@ def build_windows_kill_command(pid_file: str) -> str:
:param pid_file: Path to the PID file
:return: PowerShell command to kill the process
"""
- escaped_path = pid_file.replace("'", "''")
- script = f"""$path = '{escaped_path}'
-if (Test-Path $path) {{
- $procId = Get-Content $path
+ script = f"""$path = {_ps_path(pid_file)}
+if (Test-Path -LiteralPath $path) {{
+ $procId = Get-Content -LiteralPath $path
& taskkill.exe /PID $procId /T /F 2>$null
}}"""
script_bytes = script.encode("utf-16-le")
@@ -490,8 +551,7 @@ def build_windows_cleanup_command(job_dir: str, base_dir:
str | None = None) ->
:raises ValueError: If job_dir is not under the expected base directory
"""
_validate_job_dir(job_dir, "windows", base_dir)
- escaped_path = job_dir.replace("'", "''")
- script = f"Remove-Item -Recurse -Force -Path '{escaped_path}' -ErrorAction
SilentlyContinue"
+ script = f"Remove-Item -Recurse -Force -LiteralPath {_ps_path(job_dir)}
-ErrorAction SilentlyContinue"
script_bytes = script.encode("utf-16-le")
encoded_script = base64.b64encode(script_bytes).decode("ascii")
return f"powershell.exe -NoProfile -NonInteractive -EncodedCommand
{encoded_script}"
diff --git a/providers/ssh/tests/unit/ssh/utils/test_remote_job.py
b/providers/ssh/tests/unit/ssh/utils/test_remote_job.py
index e743c9fdd93..8e974a6769b 100644
--- a/providers/ssh/tests/unit/ssh/utils/test_remote_job.py
+++ b/providers/ssh/tests/unit/ssh/utils/test_remote_job.py
@@ -21,6 +21,7 @@ import base64
import contextlib
import os
import pty
+import re
import select
import shutil
import subprocess
@@ -47,6 +48,39 @@ from airflow.providers.ssh.utils.remote_job import (
generate_job_id,
)
+# PowerShell ends a single-quoted string at any of these; a doubled one stands
for itself.
+_PS_QUOTES = "'\u2018\u2019\u201a\u201b"
+
+
+def _decode(cmd: str) -> str:
+ return base64.b64decode(cmd.split("-EncodedCommand
")[1]).decode("utf-16-le")
+
+
+def _read_ps_single_quoted(text: str) -> tuple[str, str]:
+ """Read the PowerShell single-quoted string at the start of ``text``;
return (value, rest)."""
+ assert text[0] in _PS_QUOTES, f"not a single-quoted string: {text[:40]!r}"
+ value, i = [], 1
+ while i < len(text):
+ if text[i] in _PS_QUOTES:
+ if i + 1 < len(text) and text[i + 1] in _PS_QUOTES:
+ value.append(text[i + 1])
+ i += 2
+ continue
+ return "".join(value), text[i + 1 :]
+ value.append(text[i])
+ i += 1
+ raise AssertionError("unterminated single-quoted string")
+
+
+def _eval_ps_path(expr: str, env: dict[str, str]) -> str:
+ """Evaluate a path expression the Windows builders emit: ``'...'`` or
``($env:NAME + '...')``."""
+ if expr.startswith("($env:"):
+ name, rest = expr[len("($env:") :].split(" + ", 1)
+ value, rest = _read_ps_single_quoted(rest)
+ assert rest.startswith(")")
+ return env[name] + value
+ return _read_ps_single_quoted(expr)[0]
+
class TestGenerateJobId:
def test_generates_unique_ids(self):
@@ -159,11 +193,9 @@ class TestBuildWindowsWrapperCommand:
assert "powershell.exe" in wrapper
assert "-EncodedCommand" in wrapper
- # Decode and verify script content
- encoded_script = wrapper.split("-EncodedCommand ")[1]
- decoded_script = base64.b64decode(encoded_script).decode("utf-16-le")
+ decoded_script = _decode(wrapper)
assert "New-Item -ItemType Directory" in decoded_script
- assert "Start-Process" in decoded_script
+ assert "Win32_Process" in decoded_script
def test_with_environment(self):
"""Test Windows wrapper with environment variables."""
@@ -176,6 +208,105 @@ class TestBuildWindowsWrapperCommand:
assert wrapper is not None
assert "-EncodedCommand" in wrapper
+ def test_command_reaches_its_script_verbatim(self):
+ """The command is written to cmd.ps1 as a quoted literal, so no
character in it is special."""
+ command = "python -c \"print(1+1)\"\n$s = @'\nline\n'@\nWrite-Output
`$done \u2019q $(hostname)"
+ paths = RemoteJobPaths(job_id="test_job", remote_os="windows")
+
+ script = _decode(build_windows_wrapper_command(command, paths))
+
+ parsed, rest =
_read_ps_single_quoted(script.split('WriteAllText("$d\\cmd.ps1", ', 1)[1])
+ assert parsed == command
+ assert rest.startswith(", ")
+
+ def test_job_script_runs_without_file_so_policy_cannot_block_it(self):
+ """-File is refused under a signed-only Group Policy; the job script
is run as a script block."""
+ paths = RemoteJobPaths(job_id="test_job", remote_os="windows")
+
+ script = _decode(build_windows_wrapper_command("hostname", paths))
+
+ assert "-File" not in script
+ assert "[scriptblock]::Create([IO.File]::ReadAllText(" in script
+
+ def test_log_stays_readable_and_gets_every_stream(self):
+ """Add-Content locks the log until the command ends, and 2>&1 alone
drops Write-Host."""
+ paths = RemoteJobPaths(job_id="test_job", remote_os="windows")
+
+ script = _decode(build_windows_wrapper_command("hostname", paths))
+
+ assert "[System.IO.FileStream]::new($logPath, 'Append', 'Write',
'ReadWrite')" in script
+ assert "$log.AutoFlush = $true" in script
+ assert "Add-Content" not in script
+ pipeline = script.split('& "$b\\cmd.ps1" *>&1 |', 1)[1].split("$ec =
", 1)[0]
+ assert '[System.Management.Automation.ErrorRecord]) { "$_" }' in
pipeline
+ assert (
+ pipeline.index("ErrorRecord") < pipeline.index("Out-String
-Stream") < pipeline.index("WriteLine")
+ )
+
+ def test_environment_cannot_redirect_the_log(self):
+ """The writer opens a local path, so an environment entry named
LOG_FILE changes nothing."""
+ paths = RemoteJobPaths(job_id="test_job", remote_os="windows")
+
+ script = _decode(build_windows_wrapper_command("hostname", paths,
environment={"LOG_FILE": "C:\\x"}))
+
+ assert script.index('$logPath = "$b\\stdout.log"') <
script.index("$env:LOG_FILE = 'C:\\x'")
+ assert "FileStream]::new($logPath," in script
+
+ def test_exit_code_is_written_however_the_command_ends(self):
+ """exit, throw and parse errors all reach finally; a negative code is
recorded unsigned."""
+ paths = RemoteJobPaths(job_id="test_job", remote_os="windows")
+
+ script = _decode(build_windows_wrapper_command("hostname", paths))
+
+ before_try, rest = script.split("\ntry {\n", 1)
+ try_block, rest = rest.split("\n} catch {\n", 1)
+ catch_block, finally_block = rest.split("\n} finally {\n", 1)
+ # Fails unless the command reaches its own assignment, so a throw
cannot pass as 0.
+ assert before_try.rstrip().endswith("$ec = 1")
+ assert '& "$b\\cmd.ps1"' in try_block
+ assert "$ec = if ($null -eq $LASTEXITCODE) { 0 } else { $LASTEXITCODE
}" in try_block
+ assert "$log.WriteLine" in catch_block
+ assert finally_block.index("if ($ec -lt 0) { $ec += 4294967296 }") <
finally_block.index("exit_code")
+
+ @pytest.mark.parametrize(
+ "value",
+ [
+ pytest.param("it\u2019s $HOME and $(hostname)",
id="quotes-and-expansions"),
+ pytest.param("a\n'@\nb\n\u2019@", id="here-string-terminators"),
+ ],
+ )
+ def test_environment_value_is_literal(self, value):
+ """Values are single-quoted, typographic quotes included, so nothing
expands or ends the script."""
+ paths = RemoteJobPaths(job_id="test_job", remote_os="windows")
+
+ script = _decode(build_windows_wrapper_command("hostname", paths,
environment={"MY_VAR": value}))
+
+ parsed, rest = _read_ps_single_quoted(script.split("$env:MY_VAR = ",
1)[1])
+ assert parsed == value
+ assert rest.startswith(";")
+ job_script = script.split("@'\n", 1)[1]
+ assert re.findall(f"^[{_PS_QUOTES}]@", job_script, re.M) == ["'@"]
+
+ def test_job_starts_in_the_ssh_session_directory(self):
+ """Without a working directory, Win32_Process.Create starts the job in
System32."""
+ paths = RemoteJobPaths(job_id="test_job", remote_os="windows")
+
+ script = _decode(build_windows_wrapper_command("hostname", paths))
+
+ assert ", $PWD.ProviderPath, $null)" in script
+
+ def test_setup_failure_fails_submission(self):
+ """A failed setup step or Create must stop the wrapper before it
reports a job id."""
+ paths = RemoteJobPaths(job_id="test_job", remote_os="windows")
+
+ script = _decode(build_windows_wrapper_command("hostname", paths))
+
+ assert script.startswith("$ErrorActionPreference = 'Stop'\n")
+ assert 'trap { [Console]::Error.WriteLine("$_"); exit 1 }' in script
+ check = script.index("if ($p.ReturnValue -ne 0)")
+ assert check < script.index("exit 1", check) <
script.index("Write-Output 'test_job'")
+ assert script.index("exit 1", check) < script.index('"$d\\pid"')
+
class TestLogTailCommands:
def test_posix_log_tail(self):
@@ -195,6 +326,10 @@ class TestLogTailCommands:
decoded_script = base64.b64decode(encoded_script).decode("utf-16-le")
assert "Seek(100" in decoded_script
assert "1024" in decoded_script
+ # Printed as a string, the chunk would be re-encoded in the SSH
session's console code
+ # page, which the trigger's strict UTF-8 decoding rejects; write the
bytes instead.
+ assert "[Console]::OpenStandardOutput()" in decoded_script
+ assert "$out.Write($bytes, 0, $bytes.Length)" in decoded_script
class TestFileSizeCommands:
@@ -506,6 +641,90 @@ class TestPosixPathQuoting:
assert not marker.exists()
+class TestWindowsPathQuoting:
+ """Job paths reach PowerShell literally; only a leading ``$env:NAME`` is
expanded."""
+
+ BUILDERS = ["wrapper", "cleanup", "kill", "log_tail", "file_size",
"completion"]
+
+ @staticmethod
+ def _builders(paths):
+ """Each builder's command, the text that precedes the path in its
script, and that path."""
+ return {
+ "wrapper": (build_windows_wrapper_command("hostname", paths), "$d
= ", paths.job_dir),
+ "cleanup": (
+ build_windows_cleanup_command(paths.job_dir,
base_dir=paths.base_dir),
+ "-LiteralPath ",
+ paths.job_dir,
+ ),
+ "kill": (build_windows_kill_command(paths.pid_file), "$path = ",
paths.pid_file),
+ "log_tail": (build_windows_log_tail_command(paths.log_file, 0,
64), "$path = ", paths.log_file),
+ "file_size": (build_windows_file_size_command(paths.log_file),
"$path = ", paths.log_file),
+ "completion": (
+ build_windows_completion_check_command(paths.exit_code_file),
+ "$path = ",
+ paths.exit_code_file,
+ ),
+ }
+
+ BASE_DIRS = [
+ pytest.param(None, id="default-base-dir"),
+ pytest.param("$Env:TEMP\\jobs", id="env-prefix-any-case"),
+ pytest.param("D:\\jobs [1] $x `y \u2019q 'z' $(hostname)",
id="metacharacters"),
+ ]
+
+ @staticmethod
+ def _resolved(path: str) -> str:
+ return re.sub(r"(?i)^\$env:TEMP", lambda _: "C:\\Temp", path)
+
+ @pytest.mark.parametrize("builder", BUILDERS)
+ @pytest.mark.parametrize("base_dir", BASE_DIRS)
+ def test_path_evaluates_to_itself(self, builder, base_dir):
+ paths = RemoteJobPaths(job_id="job_123", remote_os="windows",
base_dir=base_dir)
+ cmd, prefix, path = self._builders(paths)[builder]
+
+ expr = _decode(cmd).split(prefix, 1)[1]
+
+ assert _eval_ps_path(expr, env={"TEMP": "C:\\Temp"}) ==
self._resolved(path)
+
+ @pytest.mark.parametrize("base_dir", BASE_DIRS)
+ def test_wrapper_expands_env_prefix_once(self, base_dir):
+ """The job's process gets the resolved $d, so it never re-expands
$env:NAME itself."""
+ paths = RemoteJobPaths(job_id="job_123", remote_os="windows",
base_dir=base_dir)
+
+ script = _decode(build_windows_wrapper_command("hostname", paths))
+
+ env_prefix = r"(?i)\$env:TEMP"
+ assert len(re.findall(env_prefix, script)) ==
len(re.findall(env_prefix, paths.job_dir))
+ assert "$launch = 'powershell.exe" in script
+ assert "+ $q +" in script
+
+ @pytest.mark.parametrize("base_dir", BASE_DIRS)
+ def test_launch_command_evaluates_to_the_job_dir(self, base_dir):
+ """$launch quotes the resolved $d at runtime; emulate that step and
read the result back."""
+ paths = RemoteJobPaths(job_id="job_123", remote_os="windows",
base_dir=base_dir)
+ resolved = self._resolved(paths.job_dir)
+
+ script = _decode(build_windows_wrapper_command("hostname", paths))
+
+ pattern = re.search(r"\$q = \"'\" \+ \(\$d -replace \"(.+?)\",
'\$1\$1'\) \+ \"'\"", script)
+ assert pattern
+ q = "'" + re.sub(pattern.group(1), r"\1\1", resolved) + "'"
+ head, tail = script.split("$launch = ", 1)[1].split("\n",
1)[0].split(" + $q + ")
+ launch = _read_ps_single_quoted(head)[0] + q +
_read_ps_single_quoted(tail)[0]
+ assert _read_ps_single_quoted(launch.split("$b = ", 1)[1])[0] ==
resolved
+ assert launch.endswith("ReadAllText($b + '\\job.ps1')))\"")
+
+ @pytest.mark.parametrize("builder", BUILDERS)
+ def test_paths_are_never_globbed(self, builder):
+ """``-Path``, named or positional, reads ``[`` and ``]`` as
wildcards."""
+ paths = RemoteJobPaths(job_id="job_123", remote_os="windows")
+
+ script = _decode(self._builders(paths)[builder][0])
+
+ cmdlets =
r"Test-Path|Get-Item|Get-Content|Set-Content|Add-Content|Remove-Item|Move-Item"
+ assert not re.search(rf"\b({cmdlets})\b(?![^\n]*-LiteralPath)", script)
+
+
@pytest.mark.skipif(shutil.which("bash") is None, reason="needs bash to run
the wrapper")
class TestPosixEnvironmentQuoting:
"""An environment value must reach the job verbatim and never run as shell
code."""
