goutamadwant opened a new pull request, #13249:
URL: https://github.com/apache/seatunnel/pull/13249
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code
changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB
issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the
pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix
typo in README.md doc`.
-->
### Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull
request adds checkstyle plugin.-->
Prepare Python in Apache SeaTunnel E2E engine images before starting runtime
containers.
Fixes #13248.
**Depends on #13247. Merge the G1 correction first.** The Python-only Java
17 runs completed their jobs but failed the existing cleanup assertion on
native G1 workers. The successful Java 17 results below combine both patches.
This branch contains only the Python image-preparation change.
- Replace runtime package installation in the existing Python
source/transform suites with a shared test-image helper.
- Preserve the selected engine image, platform, original USER and inherited
JDK/startup configuration. Existing Python gates and executable allowlists
remain unchanged.
- Install only when Python is absent, propagate preparation failures, cache
successful preparation per source image within the test JVM, and use normal
Testcontainers temporary-image cleanup.
- Add an offline/configuration regression to existing PythonIT and extend
Spark provider-policy coverage.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as
the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes
- provide the console output, description and/or an example to show the
behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to
the released SeaTunnel versions or within the unreleased branches such as dev.
If no, write 'No'.
If you are adding/modifying connector documents, please follow our new
specifications: https://github.com/apache/seatunnel/issues/4544.
-->
No production user-facing change.
Before: start engine container, install Python from mirrors, then run the
test. A runtime package DNS/mirror failure prevents any job from running.
After: prepare the engine-derived image with Python, start the runtime
container, verify the interpreter and run the existing job. Python execution no
longer needs runtime package-mirror access.
No public API, option, dependency, security policy or data-format changes.
The Java 11 Zeta default, Flink 1.13 skip, engine selection and container
lifecycles are preserved. Initial preparation still needs mirrors when Python
is absent. No container reuse is added.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some
test cases that check the changes thoroughly including negative and positive
cases if possible.
If it was tested in a way different from regular unit tests, please clarify
how you tested step by step, ideally copy and paste-able, so that other
reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why
it was difficult to add.
If you are adding E2E test cases, maybe refer to
https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql.conf,
here is a good example.
-->
- Reproduced the original Java 11 setup failure with runtime networking
disabled: package DNS errors, `Unable to locate package python3`, exit 100. The
same base image lacked Python with normal networking. This does not identify
the cause of any particular hosted DNS failure.
- Java 11 Zeta: source 3/3 and transform 4/4 passed.
- Java 17 Zeta with #13247: source 3/3 and transform 4/4 passed.
- Staged Flink 1.20 AMD64 image under emulation, actual Java 11: transform
4/4 passed.
- These 18 passing project JUnit cases comprise 16 engine jobs and 2
offline/configuration regressions. They ran through a standalone JUnit Platform
launcher with lifecycle callbacks, not Maven Failsafe. Validation-only Java 17
image and isolated mount substitutions were restored before commit.
- Offline regression checks Python JSON output, inherited image metadata,
equal baseline/prepared Java versions and availability of `jps`, `jstack` and
`jmap`. Java 11/17 helper controls passed. A Spark 3.5 image control preserved
nonroot USER/configuration and executed Python without runtime networking.
- Three focused Spark provider-policy cases, scoped formatting and build
verification passed. Existing source assertions retain the
three-row/ID-range/nonnull-name checks, without claiming exact per-row equality
for all values.
Pending: full hosted matrix, official Flink 1.15/1.18 runtime images and
Spark engine jobs. The staged Flink run and Spark image control do not
establish those results. Hosted CI is pending.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
Checklist applicability: no new JARs, connector registration, user
documentation or incompatible changes are introduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]