airajena opened a new issue, #68:
URL: https://github.com/apache/fineract-business-intelligence/issues/68
## Summary
Found few correctness/reliability gaps during a full live audit of the
pipeline: a nondeterministic dbt test failure, a CI/local drift risk,
non-idempotent bootstrap scripts, and a missing pipeline lock.
## Issues found
- **Nondeterministic dbt test failures** : two singular tests referenced
`fact_loan_snapshot` and raw tables as plain SQL instead of `ref()`/`source()`,
so dbt couldn't schedule them correctly relative to the model they depend on.
Caused intermittent "relation does not exist" errors depending on build thread
timing.
- **CI/local bootstrap drift** : CI's integration-test job inlined a
hand-copied, already-drifted view SQL block instead of calling
`bootstrap_source.sh`. CI and local dev could silently diverge over time.
- **Bootstrap scripts not safely re-runnable** : `bootstrap_source.sh` /
`bootstrap_fineract_source.sh` used `CREATE OR REPLACE VIEW` on
`batch_job_execution`, which fails if the view's columns need reordering,
breaking idempotency on a second run.
- **No protection against overlapping pipeline runs** : `run_pipeline.sh`
had no lock, so two concurrent invocations could collide.
- **`.gitignore` gap** : `.venv/` wasn't ignored, risking accidental commits
of a contributor's virtual environment.
## Fix
- Replace raw identifiers with `{{ ref('fact_loan_snapshot') }}` / `{{
source('raw', ...) }}` in both dbt tests
- CI now calls `bootstrap_source.sh` directly instead of an inlined copy
- Wrap bootstrap SQL in `BEGIN`/`COMMIT` and add `DROP VIEW IF EXISTS`
before recreating `batch_job_execution`
- Add a `flock`-based lock to `run_pipeline.sh`, and auto-run `dbt deps` so
a fresh clone needs no manual step
- Add `.venv/` to `.gitignore`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
