kevdoran commented on PR #10909:
URL: https://github.com/apache/nifi/pull/10909#issuecomment-3922819219
Thanks for the feedback @bbende @markap14. Here is a revised approach that
makes the complexity of asset synchronization and internal/external asset id
mapping the responsibility of the provider rather than the nifi framework.
## Summary
Extends the `ConnectorConfigurationProvider` extension point to support
externally managed connector assets (binary files such as JDBC drivers and SSL
certificates).
### Design Principle
The provider owns everything: asset storage, the mapping between NiFi asset
UUIDs and external identifiers, digest tracking for change detection, and
rollback on failure. The framework's `StandardConnectorRepository` operates
exclusively in NiFi UUID space and never knows what an external identifier is.
### Architecture
```mermaid
graph TB
subgraph rest[REST / DAO Layer]
DAO["StandardConnectorDAO"]
end
subgraph repo[Repository Layer]
REPO["StandardConnectorRepository\n(no ID translation)"]
end
subgraph local[Local Asset Storage]
CAM["StandardConnectorAssetManager\n(ID-based paths, random UUIDs)"]
end
subgraph provider[Provider Extension]
CCP["ConnectorConfigurationProvider\n(owns external ID mapping +
digests)"]
EXT["External Store"]
end
DAO -->|"storeAsset / getAsset\ndeleteAssets / syncAssetsFromProvider"|
REPO
REPO -->|"storeAsset(nifiUuid)\ndeleteAsset(nifiUuid)\nsyncAssets() /
load()"| CCP
REPO --> CAM
CCP --> EXT
CCP -->|"createAsset / saveAsset"| CAM
```
### Asset Upload Flow
```mermaid
sequenceDiagram
participant DAO as StandardConnectorDAO
participant REPO as StandardConnectorRepository
participant CCP as ConnectorConfigurationProvider
participant CAM as StandardConnectorAssetManager
DAO->>REPO: storeAsset(connectorId, nifiUuid, name, stream)
REPO->>CCP: storeAsset(connectorId, nifiUuid, name, stream)
CCP->>CAM: saveAsset(connectorId, nifiUuid, name, stream)
Note over CAM: Streams to disk at\n{connectorId}/{nifiUuid}/{name}
CCP->>CCP: open FileInputStream from saved file
CCP->>EXT: upload stream to external store
CCP->>CCP: persist nifiUuid ↔ externalId mapping\nto local state file
REPO->>CAM: getAsset(nifiUuid)
CAM-->>REPO: Asset
REPO-->>DAO: Asset
```
### On-Demand Asset Sync (applyUpdate / verifyConfigurationStep)
```mermaid
sequenceDiagram
participant REPO as StandardConnectorRepository
participant CCP as ConnectorConfigurationProvider
participant CAM as StandardConnectorAssetManager
participant EXT as External Store
REPO->>CCP: syncAssets(connectorId)
CCP->>EXT: LIST assets/ → current digests
loop for each asset in local mapping
alt digest changed or file missing
CCP->>EXT: stream binary from external store
CCP->>CAM: createAsset(connectorId, name, stream) → new UUID
CCP->>CAM: deleteAsset(oldUuid)
CCP->>CCP: update local mapping\n(new UUID + new digest)
end
end
REPO->>CCP: load(connectorId)
Note over CCP: Translates external IDs → NiFi UUIDs\nusing updated local
mapping
CCP-->>REPO: ConnectorWorkingConfiguration\n(NiFi UUIDs in assetIds)
REPO->>REPO: apply updated config\nto working context
```
### Changes
**`nifi-framework-api`**
- `ConnectorConfigurationProvider` — three new abstract methods:
`storeAsset(connectorId, nifiUuid, name, content)`, `deleteAsset(connectorId,
nifiUuid)`, `syncAssets(connectorId)`
- `ConnectorConfigurationProviderInitializationContext` — adds
`getAssetManager()` so providers can perform local asset I/O
- `ConnectorWorkingConfiguration` — stays as `name` +
`workingFlowConfiguration` only; asset ID mapping is a provider-internal concern
**`nifi-framework-core`**
- `StandardConnectorRepository` — asset methods delegate to the provider;
`syncAssetsFromProvider()` calls `provider.syncAssets()` then reloads via
`provider.load()`; `applyUpdate()` syncs assets before transitioning state; all
ID translation maps and methods removed
- `ConnectorAssetRepository` / `StandardConnectorAssetRepository` —
**deleted**; `StandardConnectorRepository` holds `AssetManager` directly
- `StandardConnectorConfigurationProviderInitializationContext` — wires in
the `AssetManager`
- `FlowController` — passes `connectorAssetManager` to the provider
initialization context
**`nifi-web-api`**
- `StandardConnectorDAO` — asset operations route through
`ConnectorRepository`; `syncAssetsFromProvider` called before
`verifyConfigurationStep`
- `DtoFactory` / `WebApplicationConfiguration` — use `ConnectorRepository`
directly for asset name lookup
### Key Invariant
> **No steady-state code path overwrites an existing asset UUID's file.**
`StandardConnectorAssetManager` uses ID-based file paths
(`{storageDir}/{connectorId}/{assetId}/{name}`) with random UUIDs, so active
and working contexts can reference the same logical asset (same filename) as
separate files under different UUIDs without collision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
