[
https://issues.apache.org/jira/browse/NIFI-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18068719#comment-18068719
]
ASF subversion and git services commented on NIFI-15538:
--------------------------------------------------------
Commit 77caec7c7554d22199bfb0dc652198fd085dcd31 in nifi's branch
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=77caec7c755 ]
NIFI-15258 Initial implementation of Connectors (#10993)
- NIFI-15259 Added REST API for Connectors
Introduced comprehensive REST endpoints for managing Connectors through
the NiFi web interface and API clients.
Key capabilities:
- Full lifecycle management (create, read, update, delete)
- Start/stop control with run status endpoints
- Multi-step configuration with property groups supporting string
literals, asset references, and secret references
- Asynchronous configuration verification with polling support
- Access to connector's managed flow and process group status
- Search functionality within connector's encapsulated flow
Cluster support:
- Request replication across cluster nodes
- Response merging for dynamic property allowable values that may
differ between nodes
- Consistent configuration step and property descriptor handling
Infrastructure:
- Audit logging for all connector configuration changes
- Authorization integration with NiFi's policy framework
- UI extension support for connector-specific interfaces
- Frontend updates for connector access policy management
- NIFI-15323 Adding operate permissions to the ConnectorEntity
- NIFI-15322 Require all property descriptors within a PropertyGroup /
ConfigurationStep are unique (#10626)
- NIFI-15322 Huge refactoring of Connectors to require that <Step Name,
Property Name> tuple is unique. Requiring that is simple, but now that we
enforce that, we can greatly simplify much
of the code.
- NIFI-15322 Added StepConfiguration to simplify API rather than using
Map<String, ConnectorValueReference> in some places
- NIFI-15322 Now that we enforce unique property names per step, introduced
StepConfigurationContext to further reduce API complexity
- NIFI-15326 Adding support of configuration step documentation. (#10631)
- NIFI-15322 Adapting to new connector configuration step model.
- NIFI-15330 Allow GhostConnector to be updated (#10632)
- NIFI-15312 Implementation of Parameter Provider based Secrets Manager;
updated mock framework to support secrets; validation cleanup (#10638)
- NIFI-15336 Created AuthorizableSecret and implemented
ParameterProviderSecretProvider to provide the Parameter Provider as the
associated Authorizable (#10644)
- NIFI-15258 Enable Custom UI to retrieve connector with flow contexts (#10652)
- NIFI-15352 Added ConnectorClient to toolkit-cli so that it can be used in
system tests; created system tests for Connectors; bug fixes found running
tests; updated Secret/SecretReference to support fully qualified names
- NIFI-15343: Adding an endpoint to return available secrets to the connector
configuration wizard.
-NIFI-15343: Adding providerId to SecretDTO.
- NIFI-15343: Skipping ParameterProviders that are invalid or validating when
fetching secrets.
- NIFI-15343: Adding fully qualified name to SecretDTO.
- NIFI-15315 Add support for assets in connectors (#10647)
- Support multiple asset references
- Clean up of assets in StandardConnectorDAO after apply and discard
- Add endpoint for listing assets for a given connector
- Add endpoint for discarding the working config of a connector
- Add endpoint for retrieving asset content
- Add system test for connectors that uploads an asset
- NIFI-15361 Allowing configuration step documentation to be returned
independent of any connector instances.
- NIFI-15369 Allow ConfigurationStep to depend on another
(ConfigurationStep,Property) tuple (#10669)
- NIFI-15367 Ensure that Connectors' implicit parameter contexts are not
registered with ParameterContextManager; added system test to verify; fixed bug
in ConnectorAssetsIT where it did not wait for connector to return to its
'STOPPED' state before attempting to delete
- NIFI-15370 Add Connector ID to process groups and do not register create
process groups if they have a connector ID. (#10673)
- NIFI-15356 Adding authorization to the StandardNiFiConnectorWebContext.
(#10660)
- NIFI-15353 Adding support for rendering Connector Documentation. (#10683)
- NIFI-15429 Adding an optional query parameter for specifying which process
group should be returned within the Connectors flow.
- NIFI-15430: Ensure that if we fail to initialize a Connector, we create a
GhostConnector instead and ensure that we also proivde the extensionMissing
flag on ConnectorNode
- NIFI-15430: Added extensionMissing flag to Connector DTO
- NIFI-15427: Added abiliy to drop flowfiles / drain flowfiles from a
Connector; added some system tests to verify existing behavior; fixed existing
issue with IT
- NIFI-15433: If connector validation throws an Exception keep trying until it
completes (with a timeout between retries); some bug fixes
- NIFI-15434: Ensure that we start or stop connectors on flow sync (#10737)
- NIFI-15445: Removed 'disabled' state from Connectors; also fixed bug to
ensure that if we fail to apply an update to a Connector that we stop it.
- NIFI-15440: Implementation of ConnectorActions (#10748)
- NIFI-15439: Ensure that Process Groups are accessible from the appropriate
endpoint but not accessible elsewhere
- NIFI-15439: FlowAnalyzingRegistryClient should use an explicit Connector ID
of null when retrieving Process Group from FlowManager
- NIFI-15468: Ensure that we properly initialize ConnectorNode even if a Ghost
Connector is created (#10775)
- NIFI-15446: When invoking ConnectorMethod, make sure to serialize/deserialize
the arguments and return value
- NIFI-15376: Adding dependencies to the configuration step dto. (#10674)
- NIFI-15453: Adding an endpoint to return all controller services for a
process group in a connector.
- NIFI-15472: Added addAsset methods to ConnectorTestRunner (#10776)
- NIFI-15482: MockConnectorAssetManager requires SPI file to start (#10783)
- NIFI-15451: Added ability for Connectors to retrieve bundles available for
component types and updated VersionedFlowUtils to make use of it to easily
update Versioned components
- NIFI-15451: Added additional unit tests to VersionedFlowUtils; updates to how
we compare component versions
- NIFI-15481: Updated ConnectorActions to reflect that starting is not allowed
when invalid. Also updated ValidationState to ensure that we don't return any
ValidationResult with a valid flag of true as a 'validation error'.
- NIFI-15461: Added ability to initiate drainage of Connector's FlowFiles and
calce; added tests to verify; some bug fixes (#10767)
- NIFI-15489: Addressed the 4 bugs called out in Jira: Wrong delimiter in
StandardConnectorPropertyValue.asList,
ConnectorValidationContextBridge.getProperty() does not resolve property value
properly; StandaloneProcessGroupLifecycle.stop() does not await controller
service disabling; switched hashmap in StandaloneProcessGroupLifecycle to
ConcurrentHashMap (#10798)
- NIFI-15490: If any processors or controller services are missing, make
Connector invalid (#10799)
- NIFI-15485: Include provider name when providing fqn for secrets (#10801)
- NIFI-15495 Restart Connectors that reference assets that were synchronized
- Ensure Connectors re-resolve property values before starting
- Ensure asset clean up happens only after applyUpdate fully finishes
- Add connector asset properties to default nifi.properties
- NIFI-15465 Allow client-specified connector component ids (#10768)
- NIFI-15467: Ensure that Connectors' versions are automatically resolved on
startup
- Allow connectors to specify Bundle Compatability when updating flow
- Default to BundleCompatability.RESOLVE_BUNDLE when connectors are updating
flows
- Bug fixes; updated MockConnectorInitializationContext to extend from
StandardConnectorInitializationContext to cut down on code duplication
- NIFI-15529: Fix to StatelessGroupLifecycle to ensure that we set the desired
state of Stateless Group Node before attempting to schedule; updated
StandardProcessScheduler.startStatelessGroup to make it more clear if this is
left out again at some point (#10832)
- NIFI-15525: Ensure that we call onStepConfigured when we restore flow; ensure
that we synchronized Parmaeter Providers before Connectors (#10835)
- NIFI-15509: Enable Component reload additional classpath resources if needed
(#10812)
- NIFI-15536: Change Connector API to use DescribedValue vs AllowableValue
(#10841)
- NIFI-15511: Added endpoints for purging all FlowFiles for a given Connector;
added method to ConnectorClient to call these endpoints; updated system tests
to use these endpoints when tearing down flows; some bug fixes
- NIFI-15514: Ensure that Parameter Contexts are assigned to all Process Groups
in a Connector and not just the top-level group. Ensure that all components are
started when Connector starts instead of just Processors and Controller Services
- When Working Context is recreated, ensure that we appropriately apply
Parameter Context to newly created Process Group(s)
- Ensure that when we cleanup unused assets for Connectors that we consider any
assets that are referenced in either the Working or Active context instead of
just the Active context
- Ensure that when we stop Process Group we call all tasks in background
threads instead of calling .thenRun which could potentially run in the
foreground thread
- NIFI-15514: Fixes around ensuring that processors/controller services are
properly configured and notified of any configuration changes when parameters
change
- Removed the updateParameterContexts from ProcessGroup.updateFlow, which was
added in a previous commit as we went a different direction for the fix
- NIFI-15538: When starting/stopping components allow specifying whether or not
the action should be recursive; code cleanup and simplification to use a
Virtual Thread to execute code sequentially instead of chaining
CompletableFutures.
- NIFI-15480: Added ability to drop FlowFiles selectively using a Predicate
Added system tests to verify selective dropping of flowfiles from a Connector.
Also made significant updates so that diagnostic bundles are captured when a
system test fails and diagnostic bundle also contains details of each
Connection's FlowFileQueue. This was necessary in aiding the debugging of the
new feature but is also something that has been missing for some time.
- NIFI-15480: Ensure that when we perform a selective drop on a FlowFileQueue
that we pass appropriate SWAP_FILE_DELETED / SWAP_FILE_RENAMED events to the
FlowFile Repository
- NIFI-15549: Adding endpoints for viewing and clearing state for components
that are managed by a Connector.
- NIFI-15557: Allow Connectors to enable a Controller Service using overridden
property values (#10862)
- NIFI-15488: Added significant number of debug log messages as well as some
info/error for connector-related events (#10803)
- NIFI-15544: If a Connector requires processors or controller services that
are unavailable, make Connector invalid but not ghosted (#10851)
- NIFI-15558: Sensitive Parameter References should resolve to value. (#10863)
- NIFI-15539: Updating the actions in FlowFileQueueResource to support
interacting with queues in flows managed by a Connector.
- NIFI-15539: Updating resource endpoint to include data policies for
connectors.
- NIFI-15565: Code cleanup; improvements to .cursorrules to provide better
instructions regarding automated tests
- NIFI-15355 Connector Configuration Repository (#10876)
- NIFI-15255 Introduce ConnectorConfigurationRepository Extension Interface
- NIFI-15559: Enforce data permissions when purging data from a Connector.
(#10867)
- NIFI-15577: Including connector id in provenance event dto.
- NIFI-15577: Fixing back navigation context when navigating to provenance from
the flow designer.
- NIFI-15581: Add support for ControllerService Mocks for Connectors. (#10885)
- NIFI-15579: When synchronizing PG when Versioned flow for stateless group
node, do not synchronize parameter contexts if group is within a Connector
(#10884)
- NIFI-15606: Connector Mock War implementation. (#10907)
- NIFI-15611: Fetchable Values should pull from current FlowContext (#10906)
- NIFI-15618: Bug fixes around handling of FlowFileActivity with child groups
and stateless groups (#10912)
- NIFI-15598: When enabling referenced controller services in a connector, skip
references from properties with unsatisfied dependencies (#10897)
- NIFI-15615: Ensure that we include necessary values in the ConnectorStatusDTO
(#10913)
- NIFI-15610 Add asset management to ConnectorConfigurationProvider (#10909)
- NIFI-15621: Allow Configuration to Validate with ValidationContext (#10916)
- NIFI-15640: NPE in the applyFlow should NOT crash the runtime. (#10931)
- NIFI-15629 Add ConnectorRequestContext (#10924)
- NIFI-15663: Update to use new ComponentHierarchyScope instead of on
facade/lifecycle classes (#10958)
- NIFI-15680: Not including global parameter contexts and parameters in
connector search results. (#10980)
- NIFI-15604 Expose VersionedExternalFlow that represents the Active/Working
flow from ConnectorTestRunner
- NIFI-15648 Resolve SECRET property values before fetchAllowableValues and
verify in working flow context
- NIFI-15714 Updating the logic to locate components within Connectors when
authorizing and populating provenance events
- NIFI-15710 Always allow node identities to read connectors
- NIFI-15727 Fix ConnectorRepository syncFromProvider bug
- NIFI-15731 Addressing failing selective drop connector system test
- NIFI-15731 Fix LoadBalanceIT race conditions exposed by connectors branch
Co-authored-by: Bob Paulin <[email protected]>
Co-authored-by: Bryan Bende <[email protected]>
Co-authored-by: Kevin Doran <[email protected]>
Co-authored-by: Matt Gilman <[email protected]>
Co-authored-by: Pierre Villard <[email protected]>
Co-authored-by: Mark Payne <[email protected]>
Signed-off-by: David Handermann <[email protected]>
> Ensure that components are started/stopped in proper order when Connector
> start/stop/drain occurs
> -------------------------------------------------------------------------------------------------
>
> Key: NIFI-15538
> URL: https://issues.apache.org/jira/browse/NIFI-15538
> Project: Apache NiFi
> Issue Type: Task
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
