goutamadwant opened a new issue, #13248:
URL: https://github.com/apache/seatunnel/issues/13248

   ### Search before asking
   
   Searched open and closed issues/PRs for `Python image`, `Python apt-get` and 
`PythonTestImage`. Reviewed related merged PRs #12964, #12263 and #12987. They 
address runner setup, transform lifecycle or close-test synchronization, not 
preparation of Python inside engine images.
   
   ### What happened
   
   `PythonIT` and `TestPythonTransformIT` install Python from package mirrors 
inside already-running engine containers. An image without Python therefore 
needs working package-repository DNS/network during test startup, before the 
connector or transform job can run.
   
   The current Temurin Java 11/17 images used in local validation lacked 
`python3`. Running the existing apt installation branch with runtime networking 
disabled produced DNS resolution failures, `Unable to locate package python3` 
and exit code 100. A default-network negative control resolved the repository 
host while Python was still absent from the base image. Hosted failures also 
show runtime package-resolution errors, but this reproduction does not 
establish their underlying DNS cause.
   
   ### SeaTunnel Version
   
   3.1.0-SNAPSHOT, dev `9cd7defe84e27c725ecf6dcf588be94f63477d4a`.
   
   ### SeaTunnel Config
   
   Existing Python source/transform E2E configurations. The source and 
transform enablement gates, executable allowlists and engine startup commands 
must remain unchanged.
   
   ### Running Command / reproduction
   
   With Docker and the current Java 11 image available:
   
   ```sh
   docker run --rm --network none eclipse-temurin:11-jdk sh -c \
     'command -v python3 || { apt-get -o Acquire::Retries=0 -o 
Acquire::http::Timeout=3 update && DEBIAN_FRONTEND=noninteractive apt-get 
install -y --no-install-recommends python3; }'
   ```
   
   The retry and timeout flags only bound the reproduction. The base image must 
lack Python and cached package indexes for the same package-resolution failure 
to occur. Image tags can change.
   
   ### Before
   
   ```text
   start engine container -> install Python from mirrors -> copy script -> run 
job
   ```
   
   Runtime mirror/DNS failure prevents the test from reaching the job.
   
   ### After / proposed fix
   
   ```text
   prepare engine-derived image with Python -> start engine container -> verify 
Python -> copy script -> run job
   ```
   
   Add a shared test-only `PythonTestImage` helper, used only by the existing 
Python suites. It derives from each selected engine image, preserves its 
platform and configured USER, and retains inherited engine/JDK settings. Python 
is installed only when missing. A failed build propagates and is not cached as 
a successful image. Successful preparation is cached per source image within 
the test JVM, with normal Testcontainers temporary-image cleanup.
   
   The new `PythonIT.testPreparedImageWithoutRuntimeNetwork` runs Python JSON 
output with runtime networking disabled and checks inherited image metadata, 
matching Java versions and all three required JDK tools (`jps`, `jstack`, 
`jmap`). Existing Spark provider/command selection is covered by three focused 
cases.
   
   ### Tests and results
   
   - Java 11 Zeta: source 3/3 and transform 4/4 passed.
   - Java 17 Zeta, with the separate G1 classifier correction: source 3/3 and 
transform 4/4 passed.
   - Flink 1.20: four transform scenarios passed using a staged AMD64 image 
under emulation, with actual Java 11. This is not validation of the full 
official image matrix.
   - Total: 18 passing JUnit cases, comprising 16 engine jobs and 2 
offline/configuration regressions. Zero failures/skips in these final runs.
   - Existing project E2E classes ran through a standalone JUnit Platform 
launcher with lifecycle callbacks, not Maven Failsafe. Validation-only Java 17 
image and isolated bind-mount substitutions were restored before committing.
   - Offline helper checks passed for Java 11/17. A Spark 3.5 image control 
preserved its nonroot USER/configuration and ran Python without runtime 
networking. No Spark engine job was executed.
   - Three focused Spark provider-policy tests, scoped formatting and build 
verification passed.
   
   Existing row/field assertions are retained. The source checks three rows, 
the expected ID range and nonnull names. This does not establish exact 
value/uniqueness equality for every row.
   
   ### Dependency
   
   Depends on G1 PR #13247. Merge that PR first. The Python-only Java 17 runs 
completed jobs but still failed the existing leak assertion on native `G1 
Refine#5`/`#6`. The successful Java 17 results above include both patches. This 
branch does not duplicate the G1 change.
   
   ### Advantages
   
   Python jobs can start without installing packages in running engine 
containers. Setup failures occur during image preparation, and both Python 
suites share the same preparation logic.
   
   ### Breaking changes and limitations
   
   No production APIs, configuration defaults, security gates, executable 
policies, data formats or dependencies change. The public Zeta image remains 
Java 11. Existing Flink 1.13 skipping and all engine-selection behavior remain 
unchanged. Container reuse remains disabled.
   
   Initial image construction still needs package mirrors when Python is 
absent. This is not a fully offline build. Official Flink 1.15/1.18 images, 
Spark engine workloads and the full hosted matrix remain unvalidated locally. 
Hosted CI is pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to