adumont opened a new issue, #8417:
URL: https://github.com/apache/hop/issues/8417
# Title
`Pipeline unit test database replacement is not used by DBLookup at runtime`
## Apache Hop version
2.19.0, local pipeline engine, Windows 11, Java version supplied by the
Apache
Hop client distribution.
## Description
A pipeline unit test can define a database replacement from a source
connection
to an isolated test connection. The test preparation log reports that the
replacement was applied, but a `DBLookup` transform still initializes and
uses
the source database connection at runtime.
This makes a unit test unsafe: source transforms can be replaced by input
data
sets and outputs can be replaced by golden data sets, but a `DBLookup` can
still
read the non-test database.
## Minimal reproduction
1. Create two database metadata connections:
- `source-db`: any reachable database connection.
- `test-h2`: an isolated H2 database with a table `lookup_data`.
2. Create a pipeline with:
- a source transform producing a key field;
- a `DBLookup` configured with connection `source-db` and table
`lookup_data`;
- an output transform.
3. Create a pipeline unit test that:
- replaces the source transform with an input data set;
- defines a database replacement `source-db` -> `test-h2`;
- replaces the output transform with a golden data set.
4. Run the unit test on the Local pipeline engine with detailed logging.
## Expected behavior
`DBLookup` must open `test-h2`. The unit test must not initialize, connect
to,
or query `source-db`.
## Actual behavior
The detailed log first reports a successful replacement, then `DBLookup`
loads
and opens the JDBC driver of the source connection:
```text
Replaced database connection 'source-db' with connection 'test-h2'
...
source-db - Loading JDBC driver class
'com.microsoft.sqlserver.jdbc.SQLServerDriver'
source-db - Opening JDBC connection...
source-db - JDBC connection opened successfully
DBLookup.0 - Connected to database...
```
The query is executed on `source-db`, not on `test-h2`.
## Why this matters
The unit-testing documentation presents database replacements as a way to
use a
test-specific connection. For pipelines with lookups this currently permits
an
apparently isolated unit test to access a production or otherwise non-test
data
source. In regulated environments this can prevent adoption of Hop unit
tests.
## Relevant implementation detail
`PipelineMetaModifier#getTestPipeline()` currently logs the replacement after
calling:
```java
sourceDatabaseMeta.replaceMeta(replacementDatabaseMeta);
```
See:
https://github.com/apache/hop/blob/main/plugins/misc/testing/src/main/java/org/apache/hop/testing/xp/PipelineMetaModifier.java
The runtime behavior above suggests that either this replacement is not
retained
by `DBLookup` metadata when the engine is initialized, or that `DBLookup`
resolves the original named connection again after the copied pipeline
metadata
has been prepared.
## Suggested acceptance test
Add an integration test with two distinct database types, for example source
SQL Server/H2 and replacement H2. Assert that a `DBLookup` in a pipeline unit
test opens the H2 driver only and reads a value available exclusively in the
H2
fixture. A test that merely checks that the replacement log message is
emitted
would not catch this regression.
## Notes
This report intentionally contains no proprietary pipeline, database URL,
credential, schema, or business data. The issue was reproduced while
preparing
a CSV-backed, H2-isolated pipeline unit test suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
