mattcasters opened a new pull request, #8625:
URL: https://github.com/apache/hop/pull/8625
Fixes #8623
### Description
This PR addresses and fixes the memory leak observed when using the caching
database (and file) execution information locations:
1. **Bounded In-Memory LRU Cache**:
- Replaced unbounded `HashMap` in `BaseCachingExecutionInfoLocation` with
an access-order `LinkedHashMap`.
- Added `maxCacheSize` metadata property and GUI widget (default: `50`,
order `"905"`), configurable in the metadata dialog.
- Reduced `maxCacheAge` default from 24 hours to 10 minutes (`600000` ms)
and updated i18n tooltips and labels.
- Implemented `enforceMaxCacheSize()` which evicts LRU entries and
persists dirty entries prior to eviction.
- Ensured `close()` clears the cache (`finally { cache.clear(); }`) and
synchronized `clearCaches()`.
2. **Eliminated Immortal Unwritten Cache Entries**:
- In `CacheEntry.isTooOld()`, fixed the condition so entries fall back to
`creationDate` when `lastRead` and `lastWritten` are null, preventing entries
from lingering indefinitely.
3. **Prevented PreparedStatement Collision & Statement Leak in
`retrieveIds()`**:
- In `CachingDatabaseExecutionInfoLocation.retrieveIds()`, collected
parent IDs into a list first and closed the outer query/`ResultSet` before
calling `loadCacheEntry()` for child executions, preventing statement overwrite
and leak on `Database.pstmt`.
- Reused a thread-safe Jackson `ObjectMapper` instance with
`HopJson.newMapper()`.
4. **Resource Safety in Core Database**:
- In `Database.insertRow(schemaName, tableName, fields, data)`, wrapped
insert execution in `try ... finally { closeInsert(); }` to ensure prepared
statements are always closed even on error.
5. **Lifecycle and Execution Info Timer Hardening**:
- In `Pipeline.fireExecutionFinishedListeners()`, ensured
`pipelineCompleted()`, extension point, and `releaseVfsNamespace()` are
guaranteed to execute even if a listener throws an exception.
- In `LocalPipelineEngine`, made `stopTransformExecutionInfoTimer()`
synchronized and idempotent, stopped the timer on startup failure in
`startThreads()`, and registered an execution-stopped listener to trigger
timer/location cleanup.
6. **Tests**:
- Added unit tests in `CachingDatabaseExecutionInfoLocationTest`:
- `lruCacheEvictionEnforcesMaxSize`: verifies LRU eviction bounds
memory at `maxCacheSize` while evicted entries remain persistent in the
database.
- `closeClearsCacheMap`: verifies memory cache is cleared on `close()`.
- `retrieveIdsWithChildrenLoadsChildrenCorrectly`: verifies child entry
retrieval works without statement collisions.
------------------------
- [x] Run `mvn clean install apache-rat:check` to make sure basic checks
pass. A more thorough check will be performed on your pull request
automatically.
- [x] If you have a group of commits related to the same change, please
squash your commits into one and force push your branch using `git rebase -i`.
- [x] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable.
To make clear that you license your contribution under the [Apache License
Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [x] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
