goutamadwant opened a new pull request, #12561:
URL: https://github.com/apache/seatunnel/pull/12561
### Purpose of this pull request
Fix #12560.
pgoutput streams only the columns a publication publishes: generated columns
are never published before PostgreSQL 18, VIRTUAL generated columns never, and
publication column lists can leave out any column. Postgres-CDC kept those
columns in the table schema, so streamed rows wrote NULL over snapshot values
and, with `schema-changes.enabled = true`, the first streamed change failed the
job with SE-07.
This removes unpublished columns from the catalog tables in
`PostgresIncrementalSourceFactory` when decoding with pgoutput:
- `pg_publication_tables.attnames` when the publication exists and covers
the table (PostgreSQL 15+, not `publication.autocreate.mode = filtered`, which
rewrites the publication without column lists).
- Otherwise generated columns from `pg_attribute.attgenerated`, keeping
STORED ones when the PostgreSQL 18 publication has `publish_generated_columns =
stored`.
- Primary key columns are never removed, snapshot-only jobs and other
decoding plugins are unchanged, and any lookup error keeps all columns.
### Does this PR introduce _any_ user-facing change?
Yes. With pgoutput, columns the publication does not publish are no longer
part of the Postgres-CDC schema, so they are not read in the snapshot and not
created in auto-created sink tables. Previously they were NULL for every
streamed change or failed the job. Documented in the PostgreSQL-CDC FAQ (en,
zh).
| Case (sink `px2` after UPDATE/INSERT) | before | after |
|---|---|---|
| PG 14 / 17, schema-changes off | NULL over snapshot value | excluded,
other columns correct |
| PG 14 / 17, schema-changes on | job FAILED (SE-07) | running, data correct
|
| PG 18 STORED or VIRTUAL, off / on | NULL / FAILED | excluded / running |
| PG 18 `publish_generated_columns = stored` | correct | correct, column
kept |
### How was this patch tested?
- Added `PostgresCDCGeneratedColumnIT` (postgres:14-alpine, schema changes
enabled). It fails on dev (streamed UPDATE never reaches the sink) and passes
with this change.
- Added `PostgresPublishedColumnsTest`. `./mvnw -pl
seatunnel-connectors-v2/connector-cdc/connector-cdc-postgres test` passes on
JDK 8 and JDK 11.
- `PostgresCDCIT#testAddColumnSchemaEvolutionWithRestore` and
`#testMultiTableWithRestore` pass locally.
- Manual runs on PostgreSQL 14.2, 17.9 and 18.6 cover the cases in the table
plus snapshot-only, a column-list publication and `publication.autocreate.mode
= filtered`.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
