SEPURI-SAI-KRISHNA opened a new pull request, #13297:
URL: https://github.com/apache/seatunnel/pull/13297
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Closes #13296.
MySQL column names are case-insensitive, so MySQL accepts DDL that names an
existing column in another case than the table defines it, for example `AFTER
NAME` for the column `name`. MySQL-CDC passed such names on as written
(`CustomAlterTableParserListener` takes them from the statement, and the
resolver parses with an empty Debezium `Tables`), and
`AlterTableSchemaEventHandler` compares names exactly. On dev:
- `ADD COLUMN ... AFTER NAME` fails the job with `SchemaEvolutionException`
since #13267 (before it, the column was silently added first);
- `DROP COLUMN Name` and `MODIFY COLUMN NAME ...` are not applied;
- `MODIFY ... AFTER AMOUNT` throws `NoSuchElementException`, and `CHANGE
COLUMN NAME ...` throws `IndexOutOfBoundsException`;
- `CHANGE COLUMN name NAME ...` renames the column to `NAME` in the schema,
while Debezium keeps `name`, so the change records no longer match it.
#13267 is right to reject an anchor the table does not have. Here the table
has it, so the fix gives the event the table's name for it, instead of
loosening the handler.
Changes:
- `AbstractSchemaChangeResolver.resolve` rewrites the existing columns each
parsed event refers to (the dropped, modified or renamed column and the `AFTER`
anchor) through a new `resolveColumnName` hook, against the source catalog
table it already receives. Columns that an earlier part of the same statement
adds or renames count for the later parts (`ADD x ..., ADD y ... AFTER X`). The
rewritten event keeps the job id, statement, dialect and `changeAfter`.
- The default `resolveColumnName` keeps the parsed name, so SQL Server,
Oracle and the other resolvers are unchanged; in a case-sensitive dialect
`"Name"` and `name` are different columns.
- `MySqlSchemaChangeResolver` overrides it: a name that is not a column is
resolved to the column that matches it ignoring case, when exactly one does. A
name that matches nothing is kept, so the handler still reports it.
- Every resolved column keeps the table's name, which is what Debezium's own
table model does: after `MODIFY COLUMN NOTE ...` Debezium's column is still
`note`, and the change records carry that name. MySQL itself renames the column
to `NOTE` here; I first followed MySQL, and the E2E then lost the values of
that column, because the rows still arrive as `note`. A test now compares the
names with Debezium's table model.
- For the same reason, a `CHANGE COLUMN` whose new name is the old one in
another case keeps the old name (a type change in it still applies).
- Sinks read the column names from the event, so they now also get the
table's names (for example the JDBC sink's `ALTER TABLE ... AFTER`).
Notes for reviewers:
- Not in this PR, because they happen with any case and are separate
changes; I will open issues for them: `CustomAlterTableParserListener` has no
handler for `RENAME COLUMN` (no event), drops a `CHANGE COLUMN` that keeps the
same name (also when it changes the type), and ignores `FIRST` / `AFTER` in
`CHANGE COLUMN`.
- Open PRs #11206, #12014, #11162 and #11912 change the same resolver files,
for other features; I read their diffs and none resolves column names. #11206
also adds a `MySqlSchemaChangeResolverTest`, so the new test here is
`MySqlSchemaChangeResolverColumnCaseTest` (as #12014 uses
`MySqlSchemaChangeResolverTruncateTest`) to avoid an add/add conflict.
### Does this PR introduce _any_ user-facing change?
Yes. With `schema-changes.enabled = true`, MySQL DDL that names an existing
column in another case is applied to that column and the job keeps running. The
MySQL-CDC page (EN and ZH) says so in its schema evolution section. No option,
default or API changes.
### How was this patch tested?
- `MySqlSchemaChangeResolverColumnCaseTest` (10 tests) runs the real
resolver on a schema change record and applies the result with
`AlterTableSchemaEventHandler`: `ADD ... AFTER NAME`, `DROP COLUMN Name`,
`MODIFY COLUMN NAME VARCHAR(128)` (checks the new type), `MODIFY ... AFTER
AMOUNT`, `CHANGE COLUMN NAME full_name`, `CHANGE COLUMN name NAME`, an anchor
added earlier in the same statement, the names carried by the event, a name
that matches nothing, and the names after each of those statements against
Debezium's `MySqlAntlrDdlParser` table model. With the resolver of dev, 9 of
the 10 fail (all but the name that matches nothing), as listed above.
-
`AbstractSchemaChangeResolverTest#testResolveColumnNamesKeepsParsedNamesByDefault`:
the base resolver returns the same events.
- I broke the change on purpose, one mutation at a time, and tests failed
each time: the MySQL lookup off (7 of the first 8 fail), no tracking within a
statement (1), a case-insensitive default in the base resolver (1), `MODIFY`
renaming the column to the name as written (2, one of them the Debezium
comparison), a case-only `CHANGE` taking the new name (2).
- `connector-cdc-base` 144 and `connector-cdc-mysql` 69 unit tests pass.
- `MysqlCDCWithSchemaChangeIT#testMysqlCdcWithColumnNamedInAnotherCase`
(new, Zeta): its own source and sink table and job; while the job runs, `add
column note ... after NAME`, `modify column NOTE ... after WEIGHT`, `change
column Description details ...`, `drop column WEIGHT` and `change column
details DETAILS ...`, each followed by rows, then an update and a delete. Sink
structure (column names compared ignoring case, see above) and rows must match
the source. The whole `MysqlCDCWithSchemaChangeIT` (7 tests) passes locally on
Zeta. In the new case the sink ran `ADD COLUMN note ... AFTER name`, `MODIFY
COLUMN note ... AFTER weight`, `CHANGE COLUMN description details ...`, `DROP
COLUMN weight` and `CHANGE COLUMN details details ...`. With the
`MySqlSchemaChangeResolver` of dev the job fails on the first statement with
`SchemaEvolutionException: ... Cannot add column 'note' after 'NAME'` and ends
`FAILED`.
- `spotless:check`, `MarkdownTest`, `git diff --check`.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. (not needed: no incompatible change)
* [ ] If you are contributing the connector code, please check that the
following files are updated: (not a new connector)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]