goutamadwant opened a new issue, #13248:
URL: https://github.com/apache/seatunnel/issues/13248
### Search before asking
Searched open and closed issues/PRs for `Python image`, `Python apt-get` and
`PythonTestImage`. Reviewed related merged PRs #12964, #12263 and #12987. They
address runner setup, transform lifecycle or close-test synchronization, not
preparation of Python inside engine images.
### What happened
`PythonIT` and `TestPythonTransformIT` install Python from package mirrors
inside already-running engine containers. An image without Python therefore
needs working package-repository DNS/network during test startup, before the
connector or transform job can run.
The current Temurin Java 11/17 images used in local validation lacked
`python3`. Running the existing apt installation branch with runtime networking
disabled produced DNS resolution failures, `Unable to locate package python3`
and exit code 100. A default-network negative control resolved the repository
host while Python was still absent from the base image. Hosted failures also
show runtime package-resolution errors, but this reproduction does not
establish their underlying DNS cause.
### SeaTunnel Version
3.1.0-SNAPSHOT, dev `9cd7defe84e27c725ecf6dcf588be94f63477d4a`.
### SeaTunnel Config
Existing Python source/transform E2E configurations. The source and
transform enablement gates, executable allowlists and engine startup commands
must remain unchanged.
### Running Command / reproduction
With Docker and the current Java 11 image available:
```sh
docker run --rm --network none eclipse-temurin:11-jdk sh -c \
'command -v python3 || { apt-get -o Acquire::Retries=0 -o
Acquire::http::Timeout=3 update && DEBIAN_FRONTEND=noninteractive apt-get
install -y --no-install-recommends python3; }'
```
The retry and timeout flags only bound the reproduction. The base image must
lack Python and cached package indexes for the same package-resolution failure
to occur. Image tags can change.
### Before
```text
start engine container -> install Python from mirrors -> copy script -> run
job
```
Runtime mirror/DNS failure prevents the test from reaching the job.
### After / proposed fix
```text
prepare engine-derived image with Python -> start engine container -> verify
Python -> copy script -> run job
```
Add a shared test-only `PythonTestImage` helper, used only by the existing
Python suites. It derives from each selected engine image, preserves its
platform and configured USER, and retains inherited engine/JDK settings. Python
is installed only when missing. A failed build propagates and is not cached as
a successful image. Successful preparation is cached per source image within
the test JVM, with normal Testcontainers temporary-image cleanup.
The new `PythonIT.testPreparedImageWithoutRuntimeNetwork` runs Python JSON
output with runtime networking disabled and checks inherited image metadata,
matching Java versions and all three required JDK tools (`jps`, `jstack`,
`jmap`). Existing Spark provider/command selection is covered by three focused
cases.
### Tests and results
- Java 11 Zeta: source 3/3 and transform 4/4 passed.
- Java 17 Zeta, with the separate G1 classifier correction: source 3/3 and
transform 4/4 passed.
- Flink 1.20: four transform scenarios passed using a staged AMD64 image
under emulation, with actual Java 11. This is not validation of the full
official image matrix.
- Total: 18 passing JUnit cases, comprising 16 engine jobs and 2
offline/configuration regressions. Zero failures/skips in these final runs.
- Existing project E2E classes ran through a standalone JUnit Platform
launcher with lifecycle callbacks, not Maven Failsafe. Validation-only Java 17
image and isolated bind-mount substitutions were restored before committing.
- Offline helper checks passed for Java 11/17. A Spark 3.5 image control
preserved its nonroot USER/configuration and ran Python without runtime
networking. No Spark engine job was executed.
- Three focused Spark provider-policy tests, scoped formatting and build
verification passed.
Existing row/field assertions are retained. The source checks three rows,
the expected ID range and nonnull names. This does not establish exact
value/uniqueness equality for every row.
### Dependency
Depends on G1 PR #13247. Merge that PR first. The Python-only Java 17 runs
completed jobs but still failed the existing leak assertion on native `G1
Refine#5`/`#6`. The successful Java 17 results above include both patches. This
branch does not duplicate the G1 change.
### Advantages
Python jobs can start without installing packages in running engine
containers. Setup failures occur during image preparation, and both Python
suites share the same preparation logic.
### Breaking changes and limitations
No production APIs, configuration defaults, security gates, executable
policies, data formats or dependencies change. The public Zeta image remains
Java 11. Existing Flink 1.13 skipping and all engine-selection behavior remain
unchanged. Container reuse remains disabled.
Initial image construction still needs package mirrors when Python is
absent. This is not a fully offline build. Official Flink 1.15/1.18 images,
Spark engine workloads and the full hosted matrix remain unvalidated locally.
Hosted CI is pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]