mattcasters opened a new issue, #8851:
URL: https://github.com/apache/hop/issues/8851
### What needs to happen?
Named VFS connections resolve variables and secrets once, then keep that
snapshot for the life of the provider. A later project variable, a parent-scope
`Set variables` value, or a `#{resolver:...}` secret is ignored until the
connection is saved. Saving calls `HopVfs.refresh()` and replaces the provider.
Editing a variable does not.
This was called out on the S3 provider in #8843. That pull request is the
right fix for the leak it targets (every resolve built a new file system and S3
client). Re-resolving on every lookup belongs in one change for every named
driver, which is this task. #8843 keeps the snapshot on purpose until this
lands.
#### What is wrong today
The provider registered for a VFS namespace holds the `IVariables` of that
namespace. Several drivers copy the resolved settings out of it at the first
use and never read it again.
Commons VFS 2.10 stores file systems in a map keyed by root name plus
`FileSystemOptions`. `FileSystemOptions.compareTo` treats equal contents as the
same key, so a newly built options object with the same resolved values still
returns the existing file system. Keeping one options instance is not what
makes that cache hit. It is what freezes a rotated key.
Three shapes, one effect:
**Options built once and reused as the cache key**
- `S3FileProvider.options` (`connectionOptions`), from #8843. Region,
endpoint, access key, secret key, session token, credentials file, profile,
cache TTL.
- `SftpConnectionFileProvider.connectionOptions`, filled by
`SftpConnectionOptions.build`. Host-key checking, known hosts, compression,
preferred authentications, key exchange, file name encoding, proxy, private
key, passphrase, and the password.
- `FtpConnectionSupport.connectionOptions`, shared by
`FtpConnectionFileProvider` and `FtpsConnectionFileProvider`, filled by
`FtpConnectionOptions.build`. Control encoding, listing parser, language, time
zone, date formats, active port range, SOCKS proxy, and the password.
**Resolved inside the first `doCreateFileSystem`, then kept because HopVfs
looks up with empty options**
- `AzureFileProvider`. Account, endpoint, SAS token, account key, cache TTL.
- `MinioFileProvider`. Endpoint host and port, access key, secret key,
region, part size, cache TTL.
- `HdfsFileProvider`. Endpoint, user, and the HTTP client `HdfsHttp` builds
from the connection.
- `SmbFileProvider`. `SmbSettings.resolve` and `SmbAuthenticators`.
- `HopWebDavFileProvider`. `HopWebDavConnectionAuth.apply` and the resolved
root URL.
- `DatabricksFileProvider`. Resolved Databricks connection name, root path,
and the REST client.
**Resolved in the constructor**
- `GoogleStorageFileProvider.setServiceAccountCredentials`, called from the
named-connection constructor. `doCreateFileSystem` always passes that
`newFileSystemOptions` instance, so a later variable change never reaches the
file system.
`GitFileProvider` resolves again on use through `GitCheckout` (URL,
revision, credentials, paths). Leave it alone unless a test shows a value
frozen on the file system. The checkout folder is already keyed by the resolved
URL, revision, and base path.
Default schemes that read a Hop config singleton (`s3://`, `gs://`,
`azure://`, Dropbox, Google Drive) are out of this task. They are not named
connections, and a variable in a project does not flow through them.
#### Required behaviour
On each file-system lookup of a named connection:
1. Resolve that connection's settings from the provider's `IVariables`.
2. When the resolved settings match the file system already cached for that
root, return that file system and its client. An unchanged variable must not
open a second client.
3. When they differ, close and remove the previous file system before the
new one is added to the provider cache. A stream that is already open keeps the
client it started with until that stream finishes. The next resolve uses the
new client. Old clients must not stay registered on the provider.
4. A `#{resolver:...}` secret is resolved on that lookup. An expensive
resolver caches inside the resolver. The provider does not keep the secret to
avoid the cost.
Saving the connection metadata still goes through `HopVfs.refresh()` and
replaces the provider. That path stays.
No new dialog and no new metadata field. The connection editors already call
`HopVfs.refresh()` on save.
#### How to implement it
Put the compare-and-close rule in one place under `core` (next to the VFS
code). Each provider still knows how to resolve its own metadata and how to
build its file system. The helper decides reuse versus replace, so the drivers
do not each grow a `connectionOptions` field.
For S3, SFTP, FTP, and FTPS: build the options on each lookup and stop
storing them on the provider. Equal contents still hit the Commons VFS cache.
On a content change, close the file system that was cached under the previous
options (`AbstractFileProvider.closeFileSystem` removes it and closes it), then
create the new one.
For Azure, MinIO, HDFS, SMB, WebDAV, Databricks, and Google Cloud Storage:
the settings live on the file system or on the options object filled at
construction, not in the caller's options. Give each a fingerprint of the
resolved settings. On a mismatch, close the current file system for that root
and create a new one. For Google Cloud Storage, run
`setServiceAccountCredentials` again when the resolved key file or key string
changes, rather than only from the constructor.
#### Tests
No live network. One test per driver above:
- Two resolves against the same provider and the same `IVariables`. Change a
variable (or a resolver result) between them. The second file system carries
the new value.
- The same two resolves with no change return the same file system. Assert
identity, or count creations.
`S3FileProviderFileSystemReuseTest` (added in #8843) already covers reuse
for `s3://` and for a named connection. Extend it with the variable-change
case. Add the same pair for the other drivers.
### Issue Priority
Priority: 2
### Issue Component
Component: VFS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
