[
https://issues.apache.org/jira/browse/FINERACT-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Chaudhary updated FINERACT-2903:
-----------------------------------------
Description:
h3. Problem
Every Liquibase changelog part must be registered with an {{<include>}} in a
master changelog ({{changelog-tenant.xml}} or a module's
{{module-changelog-master.xml}}). If that line is lost, for example while
resolving a merge conflict, the part *never runs* and *nothing fails during the
migration itself*. The failure surfaces later in unrelated tests. The same
happens when a module's {{<include>}} is lost from {{db.changelog-master.xml}}:
every part of that module silently never runs.
In PR #6124 a conflict resolution dropped the include, and every database shard
failed with:
{noformat}
Configuration property 'disallow-backdated-transactions' does not exist
{noformat}
Nothing in that output pointed back to the missing include.
h3. Why the existing checks don't catch it
* *Verify Liquibase Backward Compatibility* checks the changesets that actually
run. An unregistered part never runs.
* *Regression Safety - Database Changes* replays migrations from the base
branch and skips an unregistered part the same way.
* {{check-liquibase-ddl-safety.sh}} inspects DDL in changed files but does not
check registration.
* Liquibase's own validation only sees files reachable from the master
changelog.
h3. Proposed change
A script in the style of {{check-liquibase-ddl-safety.sh}} (bash, grep and awk
only, no new dependencies) and a workflow that runs it on PRs touching
{{db/changelog}}.
It *fails* when:
* a changelog part is not included by any master changelog
* a changelog that includes parts is not reachable from
{{db.changelog-master.xml}}, for example a module whose {{<include>}} was lost
* a file is included more than once
* an {{<include>}} points to a file that does not exist
It *warns*, without failing, when a new part reuses a number develop already
has in the same directory.
Example output:
{noformat}
ERROR
fineract-provider/src/main/resources/db/changelog/tenant/parts/0250_modify_note_length.xml:
not included by any master changelog
ERROR
fineract-investor/src/main/resources/db/changelog/tenant/module/investor/module-changelog-master.xml:
not reachable from
fineract-provider/src/main/resources/db/changelog/db.changelog-master.xml, so
its 24 part(s) never run
WARN
fineract-provider/src/main/resources/db/changelog/tenant/parts/0255_example.xml:
reuses 0255, which origin/develop already has; next free is 0256
{noformat}
On develop it passes as is (435 parts across 13 master changelogs, each
included exactly once and reachable from {{db.changelog-master.xml}}), so no
exceptions list is needed.
Discussed on the dev@ list: [CI check for Liquibase changelog
registration|https://lists.apache.org/thread/orbtvxsngl2kz3onvx9r6mdxwzdjkbso]
was:
h3. Problem
Every Liquibase changelog part must be registered with an {{<include>}} in a
master changelog ({{changelog-tenant.xml}} or a module's
{{module-changelog-master.xml}}). If that line is lost, for example while
resolving a merge conflict, the part *never runs* and *nothing fails during the
migration itself*. The failure surfaces later in unrelated tests.
In PR #6124 a conflict resolution dropped the include, and every database shard
failed with:
{noformat}
Configuration property 'disallow-backdated-transactions' does not exist
{noformat}
Nothing in that output pointed back to the missing include.
h3. Why the existing checks don't catch it
* *Verify Liquibase Backward Compatibility* checks the changesets that actually
run. An unregistered part never runs.
* *Regression Safety - Database Changes* replays migrations from the base
branch and skips an unregistered part the same way.
* {{check-liquibase-ddl-safety.sh}} inspects DDL in changed files but does not
check registration.
* Liquibase's own validation only sees files reachable from the master
changelog.
h3. Proposed change
A script in the style of {{check-liquibase-ddl-safety.sh}} (bash, grep and awk
only, no new dependencies) and a workflow that runs it on PRs touching
{{db/changelog}}.
It *fails* when:
* a changelog part is not included by any master changelog
* a changelog that includes parts is not reachable from
{{db.changelog-master.xml}}, for example a module whose {{<include>}} was lost
* a file is included more than once
* an {{<include>}} points to a file that does not exist
It *warns*, without failing, when a new part reuses a number develop already
has in the same directory.
Example output:
{noformat}
ERROR
fineract-provider/src/main/resources/db/changelog/tenant/parts/0250_modify_note_length.xml:
not included by any master changelog
WARN
fineract-provider/src/main/resources/db/changelog/tenant/parts/0255_example.xml:
reuses 0255, which origin/develop already has; next free is 0256
{noformat}
On develop it passes as is (435 parts across 13 master changelogs, each
included exactly once and reachable from {{db.changelog-master.xml}}), so no
exceptions list is needed.
Discussed on the dev@ list: [CI check for Liquibase changelog
registration|https://lists.apache.org/thread/orbtvxsngl2kz3onvx9r6mdxwzdjkbso]
Summary: Add a CI check that every Liquibase changelog part is included
once and reachable from db.changelog-master.xml (was: Add a CI check that
every Liquibase changelog part is included by a master changelog)
> Add a CI check that every Liquibase changelog part is included once and
> reachable from db.changelog-master.xml
> --------------------------------------------------------------------------------------------------------------
>
> Key: FINERACT-2903
> URL: https://issues.apache.org/jira/browse/FINERACT-2903
> Project: Apache Fineract
> Issue Type: Improvement
> Components: Build, Migration Scripts
> Reporter: Abhishek Chaudhary
> Assignee: Abhishek Chaudhary
> Priority: Major
> Fix For: 1.16.0
>
>
> h3. Problem
> Every Liquibase changelog part must be registered with an {{<include>}} in a
> master changelog ({{changelog-tenant.xml}} or a module's
> {{module-changelog-master.xml}}). If that line is lost, for example while
> resolving a merge conflict, the part *never runs* and *nothing fails during
> the migration itself*. The failure surfaces later in unrelated tests. The
> same happens when a module's {{<include>}} is lost from
> {{db.changelog-master.xml}}: every part of that module silently never runs.
> In PR #6124 a conflict resolution dropped the include, and every database
> shard failed with:
> {noformat}
> Configuration property 'disallow-backdated-transactions' does not exist
> {noformat}
> Nothing in that output pointed back to the missing include.
> h3. Why the existing checks don't catch it
> * *Verify Liquibase Backward Compatibility* checks the changesets that
> actually run. An unregistered part never runs.
> * *Regression Safety - Database Changes* replays migrations from the base
> branch and skips an unregistered part the same way.
> * {{check-liquibase-ddl-safety.sh}} inspects DDL in changed files but does
> not check registration.
> * Liquibase's own validation only sees files reachable from the master
> changelog.
> h3. Proposed change
> A script in the style of {{check-liquibase-ddl-safety.sh}} (bash, grep and
> awk only, no new dependencies) and a workflow that runs it on PRs touching
> {{db/changelog}}.
> It *fails* when:
> * a changelog part is not included by any master changelog
> * a changelog that includes parts is not reachable from
> {{db.changelog-master.xml}}, for example a module whose {{<include>}} was lost
> * a file is included more than once
> * an {{<include>}} points to a file that does not exist
> It *warns*, without failing, when a new part reuses a number develop already
> has in the same directory.
> Example output:
> {noformat}
> ERROR
> fineract-provider/src/main/resources/db/changelog/tenant/parts/0250_modify_note_length.xml:
> not included by any master changelog
> ERROR
> fineract-investor/src/main/resources/db/changelog/tenant/module/investor/module-changelog-master.xml:
> not reachable from
> fineract-provider/src/main/resources/db/changelog/db.changelog-master.xml, so
> its 24 part(s) never run
> WARN
> fineract-provider/src/main/resources/db/changelog/tenant/parts/0255_example.xml:
> reuses 0255, which origin/develop already has; next free is 0256
> {noformat}
> On develop it passes as is (435 parts across 13 master changelogs, each
> included exactly once and reachable from {{db.changelog-master.xml}}), so no
> exceptions list is needed.
> Discussed on the dev@ list: [CI check for Liquibase changelog
> registration|https://lists.apache.org/thread/orbtvxsngl2kz3onvx9r6mdxwzdjkbso]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
