goutamadwant opened a new pull request, #11989:
URL: https://github.com/apache/seatunnel/pull/11989
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code
changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB
issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the
pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix
typo in README.md doc`.
-->
### Purpose of this pull request
Related to #10753.
This adds the Google Pub/Sub source side as a separate follow-up to the
existing sink connector.
- Reads JSON or delimited text messages from an existing subscription.
- Supports Application Default Credentials, service account key files, and
the Pub/Sub emulator.
- Uses one logical subscription split and acknowledges messages only after
the containing SeaTunnel checkpoint completes.
- Reports subscriber and acknowledgement failures to the source task so
unacknowledged messages remain available for redelivery.
### Does this PR introduce _any_ user-facing change?
Yes.
Users can configure `GooglePubSub` as a streaming source for an existing
subscription. Checkpointing is required. The delivery contract is
at-least-once, so a failure between record emission and acknowledgement can
cause redelivery.
### How was this patch tested?
Added unit coverage for source configuration, factory creation, streaming
and checkpoint validation, deserialization failures, subscriber failures,
aborted and overlapping checkpoints, and acknowledgement failures.
Added Pub/Sub emulator E2E coverage that consumes a message with Zeta and
verifies that a checkpoint completes after the record is emitted. The existing
sink E2E matrix remains unchanged.
Verified with JDK 8 and JDK 11:
```shell
./mvnw -pl seatunnel-connectors-v2/connector-google-pubsub test
```
Verified connector and E2E module packaging:
```shell
./mvnw -pl
seatunnel-connectors-v2/connector-google-pubsub,seatunnel-e2e/seatunnel-connector-v2-e2e/connector-google-pubsub-e2e
-DskipTests package
```
Verified the Pub/Sub emulator E2E profile with eight passing tests:
```shell
./mvnw -B -T 1 verify -Dapi.version=1.44 -DskipUT=true -DskipIT=false
-Dlicense.skipAddThirdParty=true -Dskip.ui=true --no-snapshot-updates -pl
:connector-google-pubsub-e2e -am -Pci -rf :connector-google-pubsub-e2e
```
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md).
No new Jar binary package is added.
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. This change does not introduce an
incompatible change.
* [x] If you are contributing the connector code, please check that the
following files are updated:
1. Updated
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
with the source registration.
2. The existing
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
entry already includes `connector-google-pubsub`; no change is required.
3. The existing
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
rule already covers this connector; no change is required.
4. Added source coverage in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e).
5. The existing connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
entry already includes `connector-google-pubsub`; no change is required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
