zhiliang-wu opened a new pull request, #13304:
URL: https://github.com/apache/seatunnel/pull/13304

   ### Purpose of this pull request
   In batch synchronization pipelines and scheduled ETL jobs, upstream 
MaxCompute partitioned tables may not yet have partitions generated for a given 
execution window (e.g., due to upstream delays, non-business days, or empty 
scheduled dates).
   Currently, when a configured `partition_spec` does not exist in MaxCompute, 
the connector immediately fails the job during initialization.
   This pull request introduces the `ignore_missing_partition` option (boolean, 
default: `false`) for the MaxCompute Source connector:
   1. **Configurable Behavior**: When `ignore_missing_partition` is set to 
`true`, if the specified partition does not exist in MaxCompute, the enumerator 
warns and skips split generation (records count = 0), allowing the pipeline to 
proceed without failing.
   2. **Multi-Table & Single-Table Support**: Works for single-table mode as a 
global config, and multi-table mode (`table_list`) with support for per-table 
overrides.
   3. **Robust Error Handling**: Specifically handles `NoSuchObjectException` 
as a non-existent partition/table. Genuine connection, authentication, and 
network failures throw `MaxcomputeConnectorException` to prevent silent data 
loss.
   4. **Performance Optimization**: Restricts remote partition existence checks 
to the split enumerator initialization stage, avoiding redundant partition RPC 
checks in the reader execution path.
   5. **Schema-Aware Session Builder**: Fixes `buildDownloadSession` in 
`MaxcomputeUtil` to properly pass `schemaName` to the download session builder.
   ### Does this PR introduce _any_ user-facing change?
   Yes. Adds a new optional configuration option for the MaxCompute source 
connector:
   * **Name**: `ignore_missing_partition`
   * **Type**: `Boolean`
   * **Default**: `false`
   * **Description**: Whether to ignore missing partitions. If set to `true` 
and the specified partition does not exist, the task will not fail and will 
treat the partition as having 0 records.
   Documentation has been updated in both English and Chinese:
   * `docs/en/connectors/source/Maxcompute.md`
   * `docs/zh/connectors/source/Maxcompute.md`
   ### How was this patch tested?
   1. **Unit Tests Added**:
      * 
`MaxcomputeSourceFactoryTest#optionRuleRegistersIgnoreMissingPartition`: 
Verifies that `ignore_missing_partition` is registered in `optionRule()`.
      * 
`MaxcomputeSourceSplitTest#testComputeSplitsWithEmptyOrMissingPartitionTable`: 
Validates that missing or empty partition tables generate 0 splits while other 
tables generate splits normally.
      * `MaxcomputeUtilTest#testHasPartitionReturnsTrueWhenPartitionSpecEmpty`: 
Verifies empty partition spec behavior.
      * `MaxcomputeUtilTest#testHasPartitionDelegatesToTableHasPartition`: 
Verifies delegation to the MaxCompute Table API.
      * `MaxcomputeUtilTest#testHasPartitionThrowsOnException`: Confirms that 
unexpected network or runtime exceptions are rethrown as 
`MaxcomputeConnectorException`.
      * 
`MaxcomputeUtilTest#testHasPartitionReturnsFalseOnNoSuchObjectException`: 
Confirms `NoSuchObjectException` returns `false`.
   2. **Local Build & Verification**:
      * `./mvnw spotless:apply -pl :connector-maxcompute` (passed)
      * `./mvnw test -pl :connector-maxcompute` (all 95 tests passed)
      * `./mvnw -q -DskipTests verify -pl :connector-maxcompute` (passed)
   ### Check list
   * [ ] If any new Jar binary package adding in your PR, please add License 
Notice according [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [x] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the 
following files are updated:
     1. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it
     2. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in 
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in 
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector 
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to