zhiliang-wu opened a new pull request, #13304:
URL: https://github.com/apache/seatunnel/pull/13304
### Purpose of this pull request
In batch synchronization pipelines and scheduled ETL jobs, upstream
MaxCompute partitioned tables may not yet have partitions generated for a given
execution window (e.g., due to upstream delays, non-business days, or empty
scheduled dates).
Currently, when a configured `partition_spec` does not exist in MaxCompute,
the connector immediately fails the job during initialization.
This pull request introduces the `ignore_missing_partition` option (boolean,
default: `false`) for the MaxCompute Source connector:
1. **Configurable Behavior**: When `ignore_missing_partition` is set to
`true`, if the specified partition does not exist in MaxCompute, the enumerator
warns and skips split generation (records count = 0), allowing the pipeline to
proceed without failing.
2. **Multi-Table & Single-Table Support**: Works for single-table mode as a
global config, and multi-table mode (`table_list`) with support for per-table
overrides.
3. **Robust Error Handling**: Specifically handles `NoSuchObjectException`
as a non-existent partition/table. Genuine connection, authentication, and
network failures throw `MaxcomputeConnectorException` to prevent silent data
loss.
4. **Performance Optimization**: Restricts remote partition existence checks
to the split enumerator initialization stage, avoiding redundant partition RPC
checks in the reader execution path.
5. **Schema-Aware Session Builder**: Fixes `buildDownloadSession` in
`MaxcomputeUtil` to properly pass `schemaName` to the download session builder.
### Does this PR introduce _any_ user-facing change?
Yes. Adds a new optional configuration option for the MaxCompute source
connector:
* **Name**: `ignore_missing_partition`
* **Type**: `Boolean`
* **Default**: `false`
* **Description**: Whether to ignore missing partitions. If set to `true`
and the specified partition does not exist, the task will not fail and will
treat the partition as having 0 records.
Documentation has been updated in both English and Chinese:
* `docs/en/connectors/source/Maxcompute.md`
* `docs/zh/connectors/source/Maxcompute.md`
### How was this patch tested?
1. **Unit Tests Added**:
*
`MaxcomputeSourceFactoryTest#optionRuleRegistersIgnoreMissingPartition`:
Verifies that `ignore_missing_partition` is registered in `optionRule()`.
*
`MaxcomputeSourceSplitTest#testComputeSplitsWithEmptyOrMissingPartitionTable`:
Validates that missing or empty partition tables generate 0 splits while other
tables generate splits normally.
* `MaxcomputeUtilTest#testHasPartitionReturnsTrueWhenPartitionSpecEmpty`:
Verifies empty partition spec behavior.
* `MaxcomputeUtilTest#testHasPartitionDelegatesToTableHasPartition`:
Verifies delegation to the MaxCompute Table API.
* `MaxcomputeUtilTest#testHasPartitionThrowsOnException`: Confirms that
unexpected network or runtime exceptions are rethrown as
`MaxcomputeConnectorException`.
*
`MaxcomputeUtilTest#testHasPartitionReturnsFalseOnNoSuchObjectException`:
Confirms `NoSuchObjectException` returns `false`.
2. **Local Build & Verification**:
* `./mvnw spotless:apply -pl :connector-maxcompute` (passed)
* `./mvnw test -pl :connector-maxcompute` (all 95 tests passed)
* `./mvnw -q -DskipTests verify -pl :connector-maxcompute` (passed)
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according [New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]