This is an automated email from the ASF dual-hosted git repository. exceptionfactory pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/nifi.git
commit 1d2b755edf428d465d50c4025824f636bc9c7c50 Author: Joseph Witt <joew...@apache.org> AuthorDate: Thu Jul 4 19:37:16 2024 -0700 NIFI-13511 Removed ShellUserGroupProvider from NiFi and Registry This closes #9046 Signed-off-by: David Handermann <exceptionfact...@apache.org> --- nifi-code-coverage/pom.xml | 5 - .../src/main/asciidoc/administration-guide.adoc | 24 +- .../nifi-framework-nar-bom/pom.xml | 10 - nifi-framework-bundle/nifi-framework-nar/pom.xml | 5 - .../src/main/resources/conf/authorizers.xml | 25 - .../nifi-framework/nifi-shell-authorizer/pom.xml | 38 -- .../nifi/authorization/NssShellCommands.java | 56 -- .../nifi/authorization/OsxShellCommands.java | 56 -- .../nifi/authorization/RemoteShellCommands.java | 62 -- .../nifi/authorization/ShellCommandsProvider.java | 67 -- .../nifi/authorization/ShellUserGroupProvider.java | 672 --------------------- .../nifi/authorization/util/ShellRunner.java | 123 ---- ...org.apache.nifi.authorization.UserGroupProvider | 15 - nifi-framework-bundle/nifi-framework/pom.xml | 1 - .../authorization/shell/NssShellCommands.java | 55 -- .../authorization/shell/OsxShellCommands.java | 55 -- .../authorization/shell/ShellCommandsProvider.java | 67 -- .../security/authorization/shell/ShellRunner.java | 123 ---- .../shell/ShellUserGroupProvider.java | 645 -------------------- ...gistry.security.authorization.UserGroupProvider | 3 +- .../src/main/resources/conf/authorizers.xml | 21 - 21 files changed, 2 insertions(+), 2126 deletions(-) diff --git a/nifi-code-coverage/pom.xml b/nifi-code-coverage/pom.xml index ceaf8df8a3..00bda5522f 100644 --- a/nifi-code-coverage/pom.xml +++ b/nifi-code-coverage/pom.xml @@ -596,11 +596,6 @@ <artifactId>nifi-runtime</artifactId> <version>2.0.0-SNAPSHOT</version> </dependency> - <dependency> - <groupId>org.apache.nifi</groupId> - <artifactId>nifi-shell-authorizer</artifactId> - <version>2.0.0-SNAPSHOT</version> - </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-site-to-site</artifactId> diff --git a/nifi-docs/src/main/asciidoc/administration-guide.adoc b/nifi-docs/src/main/asciidoc/administration-guide.adoc index f8d1a8a469..0b2a0e5ebf 100644 --- a/nifi-docs/src/main/asciidoc/administration-guide.adoc +++ b/nifi-docs/src/main/asciidoc/administration-guide.adoc @@ -775,28 +775,6 @@ The LdapUserGroupProvider has the following properties: NOTE: Any identity mapping rules specified in _nifi.properties_ will also be applied to the user identities. Group names are not mapped. -==== ShellUserGroupProvider - -The ShellUserGroupProvider fetches user and group details from Unix-like systems using shell commands. - -This provider executes various shell pipelines with commands such as `getent` on Linux and `dscl` on macOS. - -Supported systems may be configured to retrieve users and groups from an external source, such as LDAP or NIS. In these cases the shell commands -will return those external users and groups. This provides administrators another mechanism to integrate user and group directory services. - -The ShellUserGroupProvider has the following properties: - -[options="header,footer"] -|================================================================================================================================================== -| Property Name | Description -|`Initial Refresh Delay` | Duration of initial delay before first user and group refresh. (i.e. `10 secs`). Default is `5 mins`. -|`Refresh Delay` | Duration of delay between each user and group refresh. (i.e. `10 secs`). Default is `5 mins`. -|`Exclude Groups` | Regular expression used to exclude groups. Default is '', which means no groups are excluded. -|`Exclude Users` | Regular expression used to exclude users. Default is '', which means no users are excluded. -|================================================================================================================================================== - -Like LdapUserGroupProvider, the ShellUserGroupProvider is commented out in the _authorizers.xml_ file. Refer to that comment for usage examples. - ==== AzureGraphUserGroupProvider The AzureGraphUserGroupProvider fetches users and groups from Azure Active Directory (AAD) using the Microsoft Graph API. @@ -831,7 +809,7 @@ The AzureGraphUserGroupProvider has the following properties: |`Claim for Username` | The property of the user directory object mapped to the NiFi user name field. Default is 'upn'. 'email' is another option when `nifi.security.user.oidc.fallback.claims.identifying.user` is set to 'upn'. |================================================================================================================================================== -Like LdapUserGroupProvider and ShellUserGroupProvider, the AzureGraphUserGroupProvider configuration is commented out in the _authorizers.xml_ file. Refer to the comment for a starter configuration. +Like LdapUserGroupProvider, the AzureGraphUserGroupProvider configuration is commented out in the _authorizers.xml_ file. Refer to the comment for a starter configuration. ==== Composite Implementations diff --git a/nifi-framework-bundle/nifi-framework-nar-bom/pom.xml b/nifi-framework-bundle/nifi-framework-nar-bom/pom.xml index 65e3202b6a..3aa2756dd6 100644 --- a/nifi-framework-bundle/nifi-framework-nar-bom/pom.xml +++ b/nifi-framework-bundle/nifi-framework-nar-bom/pom.xml @@ -63,12 +63,6 @@ <version>2.0.0-SNAPSHOT</version> <scope>provided</scope> </dependency> - <dependency> - <groupId>org.apache.nifi</groupId> - <artifactId>nifi-shell-authorizer</artifactId> - <version>2.0.0-SNAPSHOT</version> - <scope>provided</scope> - </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-authorizer</artifactId> @@ -248,10 +242,6 @@ <groupId>org.apache.nifi</groupId> <artifactId>nifi-file-authorizer</artifactId> </dependency> - <dependency> - <groupId>org.apache.nifi</groupId> - <artifactId>nifi-shell-authorizer</artifactId> - </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-authorizer</artifactId> diff --git a/nifi-framework-bundle/nifi-framework-nar/pom.xml b/nifi-framework-bundle/nifi-framework-nar/pom.xml index a9ac2a114e..19a6f2ba8e 100644 --- a/nifi-framework-bundle/nifi-framework-nar/pom.xml +++ b/nifi-framework-bundle/nifi-framework-nar/pom.xml @@ -55,11 +55,6 @@ <artifactId>nifi-file-authorizer</artifactId> <scope>compile</scope> </dependency> - <dependency> - <groupId>org.apache.nifi</groupId> - <artifactId>nifi-shell-authorizer</artifactId> - <scope>compile</scope> - </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-authorizer</artifactId> diff --git a/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/authorizers.xml b/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/authorizers.xml index fdaf70dc9e..190a79f589 100644 --- a/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/authorizers.xml +++ b/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/authorizers.xml @@ -165,31 +165,6 @@ </userGroupProvider> To enable the ldap-user-group-provider remove 2 lines. This is 2 of 2. --> - <!-- - The ShellUserGroupProvider provides support for retrieving users and groups by way of shell commands - on systems that support `sh`. Implementations available for Linux and Mac OS, and are selected by the - provider based on the system property `os.name`. - - 'Refresh Delay' - duration to wait between subsequent refreshes. Default is '5 mins'. - 'Exclude Groups' - regular expression used to exclude groups. Default is '', which means no groups are excluded. - 'Exclude Users' - regular expression used to exclude users. Default is '', which means no users are excluded. - 'Legacy Identifier Mode' - preserves the legacy behavior for id generation. Disabling this will ensure that - user and group ids are differentiated to handle the case where a user and group have - the same identity. Default is 'true', which means users and groups are not differentiated. - 'Command Timeout' - amount of time to wait while executing a command before timing out - --> - <!-- To enable the shell-user-group-provider remove 2 lines. This is 1 of 2. - <userGroupProvider> - <identifier>shell-user-group-provider</identifier> - <class>org.apache.nifi.authorization.ShellUserGroupProvider</class> - <property name="Refresh Delay">5 mins</property> - <property name="Exclude Groups"></property> - <property name="Exclude Users"></property> - <property name="Legacy Identifier Mode">true</property> - <property name="Command Timeout">60 seconds</property> - </userGroupProvider> - To enable the shell-user-group-provider remove 2 lines. This is 2 of 2. --> - <!-- The AzureGraphUserGroupProvider fetches users and groups from Azure Active Directory (AAD) using the Microsoft Graph API. diff --git a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/pom.xml b/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/pom.xml deleted file mode 100644 index 5e56747194..0000000000 --- a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/pom.xml +++ /dev/null @@ -1,38 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---> -<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> - <parent> - <artifactId>nifi-framework</artifactId> - <groupId>org.apache.nifi</groupId> - <version>2.0.0-SNAPSHOT</version> - </parent> - <modelVersion>4.0.0</modelVersion> - <artifactId>nifi-shell-authorizer</artifactId> - <dependencies> - - - <dependency> - <groupId>org.apache.nifi</groupId> - <artifactId>nifi-framework-api</artifactId> - </dependency> - <dependency> - <groupId>org.apache.nifi</groupId> - <artifactId>nifi-utils</artifactId> - <version>2.0.0-SNAPSHOT</version> - </dependency> - - </dependencies> -</project> diff --git a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/NssShellCommands.java b/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/NssShellCommands.java deleted file mode 100644 index c5c3f61ef9..0000000000 --- a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/NssShellCommands.java +++ /dev/null @@ -1,56 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.authorization; - - -/** - * Provides shell commands to read users and groups on NSS-enabled systems. - * - * See `man 5 nsswitch.conf` for more info. - */ -class NssShellCommands implements ShellCommandsProvider { - /** - * @return Shell command string that will return a list of users. - */ - public String getUsersList() { - return "getent passwd | cut -f 1,3,4 -d ':'"; - } - - /** - * @return Shell command string that will return a list of groups. - */ - public String getGroupsList() { - return "getent group | cut -f 1,3 -d ':'"; - } - - /** - * @param groupName name of group. - * @return Shell command string that will return a list of users for a group. - */ - public String getGroupMembers(String groupName) { - return String.format("getent group %s | cut -f 4 -d ':'", groupName); - } - - /** - * This gives exit code 0 on all tested distributions. - * - * @return Shell command string that will exit normally (0) on a suitable system. - */ - public String getSystemCheck() { - return "getent --version"; - } -} diff --git a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/OsxShellCommands.java b/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/OsxShellCommands.java deleted file mode 100644 index 90f405f428..0000000000 --- a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/OsxShellCommands.java +++ /dev/null @@ -1,56 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.authorization; - - -/** - * Provides shell commands to read users and groups on Mac OSX systems. - * - * See `man dscl` for more info. - */ -class OsxShellCommands implements ShellCommandsProvider { - /** - * @return Shell command string that will return a list of users. - */ - public String getUsersList() { - return "dscl . -readall /Users UniqueID PrimaryGroupID | awk 'BEGIN { OFS = \":\"; ORS=\"\\n\"; i=0;} /RecordName: / {name = $2;i = 0;}" + - "/PrimaryGroupID: / {gid = $2;} /^ / {if (i == 0) { i++; name = $1;}} /UniqueID: / {uid = $2;print name, uid, gid;}' | grep -v ^_"; - } - - /** - * @return Shell command string that will return a list of groups. - */ - public String getGroupsList() { - return "dscl . -list /Groups PrimaryGroupID | grep -v '^_' | sed 's/ \\{1,\\}/:/g'"; - } - - /** - * - * @param groupName name of group. - * @return Shell command string that will return a list of users for a group. - */ - public String getGroupMembers(String groupName) { - return String.format("dscl . -read /Groups/%s GroupMembership | cut -f 2- -d ' ' | sed 's/\\ /,/g'", groupName); - } - - /** - * @return Shell command string that will exit normally (0) on a suitable system. - */ - public String getSystemCheck() { - return "which dscl"; - } -} diff --git a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/RemoteShellCommands.java b/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/RemoteShellCommands.java deleted file mode 100644 index 9a6d7fd11a..0000000000 --- a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/RemoteShellCommands.java +++ /dev/null @@ -1,62 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.authorization; - - -class RemoteShellCommands implements ShellCommandsProvider { - // Carefully crafted command replacement string: - private final static String remoteCommand = "ssh " + - "-o 'StrictHostKeyChecking no' " + - "-o 'PasswordAuthentication no' " + - "-o \"RemoteCommand %s\" " + - "-i %s -p %s -l root %s"; - - private ShellCommandsProvider innerProvider; - private String privateKeyPath; - private String remoteHost; - private Integer remotePort; - - private RemoteShellCommands() { - } - - public static ShellCommandsProvider wrapOtherProvider(ShellCommandsProvider otherProvider, String keyPath, String host, Integer port) { - RemoteShellCommands remote = new RemoteShellCommands(); - - remote.innerProvider = otherProvider; - remote.privateKeyPath = keyPath; - remote.remoteHost = host; - remote.remotePort = port; - - return remote; - } - - public String getUsersList() { - return String.format(remoteCommand, innerProvider.getUsersList(), privateKeyPath, remotePort, remoteHost); - } - - public String getGroupsList() { - return String.format(remoteCommand, innerProvider.getGroupsList(), privateKeyPath, remotePort, remoteHost); - } - - public String getGroupMembers(String groupName) { - return String.format(remoteCommand, innerProvider.getGroupMembers(groupName), privateKeyPath, remotePort, remoteHost); - } - - public String getSystemCheck() { - return String.format(remoteCommand, innerProvider.getSystemCheck(), privateKeyPath, remotePort, remoteHost); - } -} diff --git a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/ShellCommandsProvider.java b/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/ShellCommandsProvider.java deleted file mode 100644 index 2879057e2c..0000000000 --- a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/ShellCommandsProvider.java +++ /dev/null @@ -1,67 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.authorization; - -/** - * Common interface for shell command strings to read users and groups. - * - */ -interface ShellCommandsProvider { - /** - * Gets the command for listing users. - * - * When executed, this command should output one record per line in this format: - * - * `username:user-id:primary-group-id` - * - * @return Shell command string that will return a list of users. - */ - String getUsersList(); - - /** - * Gets the command for listing groups. - * - * When executed, this command should output one record per line in this format: - * - * `group-name:group-id` - * - * @return Shell command string that will return a list of groups. - */ - String getGroupsList(); - - /** - * Gets the command for listing the members of a group. - * - * When executed, this command should output one line in this format: - * - * `user-name-1,user-name-2,user-name-n` - * - * @param groupName name of group. - * @return Shell command string that will return a list of users for a group. - */ - String getGroupMembers(String groupName); - - /** - * Gets the command for checking the suitability of the host system. - * - * The command is expected to exit with status 0 (zero) to indicate success, and any other status - * to indicate failure. - * - * @return Shell command string that will exit normally (0) on a suitable system. - */ - String getSystemCheck(); -} diff --git a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/ShellUserGroupProvider.java b/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/ShellUserGroupProvider.java deleted file mode 100644 index bd51f6676e..0000000000 --- a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/ShellUserGroupProvider.java +++ /dev/null @@ -1,672 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.authorization; - -import org.apache.nifi.authorization.exception.AuthorizationAccessException; -import org.apache.nifi.authorization.exception.AuthorizerCreationException; -import org.apache.nifi.authorization.exception.AuthorizerDestructionException; -import org.apache.nifi.authorization.util.ShellRunner; -import org.apache.nifi.components.PropertyValue; -import org.apache.nifi.util.FormatUtils; -import org.apache.nifi.util.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.IOException; -import java.util.Comparator; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.TreeSet; -import java.util.concurrent.Executors; -import java.util.concurrent.ScheduledExecutorService; -import java.util.concurrent.TimeUnit; -import java.util.regex.Pattern; -import java.util.regex.PatternSyntaxException; - - -/* - * ShellUserGroupProvider implements UserGroupProvider by way of shell commands. - */ -public class ShellUserGroupProvider implements UserGroupProvider { - private final static Logger logger = LoggerFactory.getLogger(ShellUserGroupProvider.class); - - private final static String OS_TYPE_ERROR = "Unsupported operating system."; - private final static String SYS_CHECK_ERROR = "System check failed - cannot provide users and groups."; - private final static Map<String, User> usersById = new HashMap<>(); // id == identifier - private final static Map<String, User> usersByName = new HashMap<>(); // name == identity - private final static Map<String, Group> groupsById = new HashMap<>(); - private final static Map<String, Group> groupsByName = new HashMap<>(); - - public static final String REFRESH_DELAY_PROPERTY = "Refresh Delay"; - private static final long MINIMUM_SYNC_INTERVAL_MILLISECONDS = 10_000; - - public static final String EXCLUDE_USER_PROPERTY = "Exclude Users"; - public static final String EXCLUDE_GROUP_PROPERTY = "Exclude Groups"; - public static final String LEGACY_IDENTIFIER_MODE = "Legacy Identifier Mode"; - public static final String COMMAND_TIMEOUT_PROPERTY = "Command Timeout"; - - private static final String DEFAULT_COMMAND_TIMEOUT = "60 seconds"; - - private long fixedDelay; - private Pattern excludeUsers; - private Pattern excludeGroups; - private boolean legacyIdentifierMode; - private int timeoutSeconds; - - // Our scheduler has one thread for users, one for groups: - private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(2); - - // Commands selected during initialization: - private ShellCommandsProvider selectedShellCommands; - - private ShellRunner shellRunner; - - // Start of the UserGroupProvider implementation. Javadoc strings - // copied from the interface definition for reference. - - /** - * Retrieves all users. Must be non null - * - * @return a list of users - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public Set<User> getUsers() throws AuthorizationAccessException { - synchronized (usersById) { - logger.debug("getUsers has user set of size: {}", usersById.size()); - return new HashSet<>(usersById.values()); - } - } - - /** - * Retrieves the user with the given identifier. - * - * @param identifier the id of the user to retrieve - * @return the user with the given id, or null if no matching user was found - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public User getUser(String identifier) throws AuthorizationAccessException { - User user; - - synchronized (usersById) { - user = usersById.get(identifier); - } - - if (user == null) { - logger.debug("getUser (by id) user not found: {}", identifier); - } else { - logger.debug("getUser (by id) found user: {} for id: {}", user, identifier); - } - return user; - } - - /** - * Retrieves the user with the given identity. - * - * @param identity the identity of the user to retrieve - * @return the user with the given identity, or null if no matching user was found - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public User getUserByIdentity(String identity) throws AuthorizationAccessException { - User user; - - synchronized (usersByName) { - user = usersByName.get(identity); - } - - if (user == null) { - logger.debug("getUser (by name) user not found: {}", identity); - } else { - logger.debug("getUser (by name) found user: {} for name: {}", user.getIdentity(), identity); - } - return user; - } - - /** - * Retrieves all groups. Must be non null - * - * @return a list of groups - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public Set<Group> getGroups() throws AuthorizationAccessException { - synchronized (groupsById) { - logger.debug("getGroups has group set of size: {}", groupsById.size()); - return new HashSet<>(groupsById.values()); - } - } - - /** - * Retrieves a Group by Id. - * - * @param identifier the identifier of the Group to retrieve - * @return the Group with the given identifier, or null if no matching group was found - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public Group getGroup(String identifier) throws AuthorizationAccessException { - Group group; - - synchronized (groupsById) { - group = groupsById.get(identifier); - } - - if (group == null) { - logger.debug("getGroup (by id) group not found: {}", identifier); - } else { - logger.debug("getGroup (by id) found group: {} for id: {}", group.getName(), identifier); - } - return group; - - } - - @Override - public Group getGroupByName(String name) throws AuthorizationAccessException { - Group group; - - synchronized (groupsByName) { - group = groupsByName.get(name); - } - - if (group == null) { - logger.debug("getGroup (by name) group not found: {}", name); - } else { - logger.debug("getGroup (by name) found group: {} for name: {}", group.getName(), name); - } - - return group; - } - - /** - * Gets a user and their groups. - * - * @return the UserAndGroups for the specified identity - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public UserAndGroups getUserAndGroups(String identity) throws AuthorizationAccessException { - User user = getUserByIdentity(identity); - logger.debug("Retrieved user {} for identity {}", user, identity); - - Set<Group> groups = new HashSet<>(); - if (user != null) { - for (Group g : getGroups()) { - if (g.getUsers().contains(user.getIdentifier())) { - logger.debug("User {} belongs to group {}", user.getIdentity(), g.getName()); - groups.add(g); - } - } - } - - if (groups.isEmpty()) { - logger.debug("User {} belongs to no groups", user); - } - - return new UserAndGroups() { - @Override - public User getUser() { - return user; - } - - @Override - public Set<Group> getGroups() { - return groups; - } - }; - } - - /** - * Called immediately after instance creation for implementers to perform additional setup - * - * @param initializationContext in which to initialize - */ - @Override - public void initialize(UserGroupProviderInitializationContext initializationContext) throws AuthorizerCreationException { - } - - /** - * Called to configure the Authorizer. - * - * @param configurationContext at the time of configuration - * @throws AuthorizerCreationException for any issues configuring the provider - */ - @Override - public void onConfigured(AuthorizerConfigurationContext configurationContext) throws AuthorizerCreationException { - logger.info("Configuring ShellUserGroupProvider"); - - fixedDelay = getDelayProperty(configurationContext, REFRESH_DELAY_PROPERTY, "5 mins"); - timeoutSeconds = getTimeoutProperty(configurationContext, COMMAND_TIMEOUT_PROPERTY, DEFAULT_COMMAND_TIMEOUT); - shellRunner = new ShellRunner(timeoutSeconds); - logger.debug("Configured ShellRunner with command timeout of '{}' seconds", timeoutSeconds); - - // Our next init step is to select the command set based on the operating system name: - ShellCommandsProvider commands = getCommandsProvider(); - - if (commands == null) { - commands = getCommandsProviderFromName(null); - setCommandsProvider(commands); - } - - // Our next init step is to run the system check from that command set to determine if the other commands - // will work on this host or not. - try { - shellRunner.runShell(commands.getSystemCheck(), "Supported System Check"); - } catch (final Exception e) { - logger.error("initialize exception: system check command: {}", commands.getSystemCheck(), e); - throw new AuthorizerCreationException(SYS_CHECK_ERROR, e); - } - - // The next step is to add the user and group exclude regexes: - try { - excludeGroups = Pattern.compile(getProperty(configurationContext, EXCLUDE_GROUP_PROPERTY, "")); - excludeUsers = Pattern.compile(getProperty(configurationContext, EXCLUDE_USER_PROPERTY, "")); - } catch (final PatternSyntaxException e) { - throw new AuthorizerCreationException(e); - } - - // Get the value for Legacy Identifier Mo - legacyIdentifierMode = Boolean.parseBoolean(getProperty(configurationContext, LEGACY_IDENTIFIER_MODE, "true")); - - // With our command set selected, and our system check passed, we can pull in the users and groups: - refreshUsersAndGroups(); - - // And finally, our last init step is to fire off the refresh thread: - scheduler.scheduleWithFixedDelay(() -> { - try { - refreshUsersAndGroups(); - } catch (final Throwable t) { - logger.error("", t); - } - }, fixedDelay, fixedDelay, TimeUnit.MILLISECONDS); - - logger.info("Completed configuration of ShellUserGroupProvider"); - } - - private static ShellCommandsProvider getCommandsProviderFromName(String osName) { - if (osName == null) { - osName = System.getProperty("os.name"); - } - - ShellCommandsProvider commands; - if (osName.startsWith("Linux")) { - logger.debug("Selected Linux command set."); - commands = new NssShellCommands(); - } else if (osName.startsWith("Mac OS X")) { - logger.debug("Selected OSX command set."); - commands = new OsxShellCommands(); - } else { - throw new AuthorizerCreationException(OS_TYPE_ERROR); - } - return commands; - } - - private String getProperty(AuthorizerConfigurationContext authContext, String propertyName, String defaultValue) { - final PropertyValue property = authContext.getProperty(propertyName); - final String value; - - if (property != null && property.isSet()) { - value = property.getValue(); - } else { - value = defaultValue; - } - return value; - - } - - private long getDelayProperty(AuthorizerConfigurationContext authContext, String propertyName, String defaultValue) { - final PropertyValue intervalProperty = authContext.getProperty(propertyName); - final String propertyValue; - final long syncInterval; - - if (intervalProperty.isSet()) { - propertyValue = intervalProperty.getValue(); - } else { - propertyValue = defaultValue; - } - - try { - syncInterval = Math.round(FormatUtils.getPreciseTimeDuration(propertyValue, TimeUnit.MILLISECONDS)); - } catch (final IllegalArgumentException ignored) { - throw new AuthorizerCreationException(String.format("The %s '%s' is not a valid time interval.", propertyName, propertyValue)); - } - - if (syncInterval < MINIMUM_SYNC_INTERVAL_MILLISECONDS) { - throw new AuthorizerCreationException(String.format("The %s '%s' is below the minimum value of '%d ms'", propertyName, propertyValue, MINIMUM_SYNC_INTERVAL_MILLISECONDS)); - } - return syncInterval; - } - - private int getTimeoutProperty(AuthorizerConfigurationContext authContext, String propertyName, String defaultValue) { - final PropertyValue timeoutProperty = authContext.getProperty(propertyName); - - final String propertyValue; - if (timeoutProperty.isSet()) { - propertyValue = timeoutProperty.getValue(); - } else { - propertyValue = defaultValue; - } - - final long timeoutValue; - try { - timeoutValue = Math.round(FormatUtils.getPreciseTimeDuration(propertyValue, TimeUnit.SECONDS)); - } catch (final IllegalArgumentException ignored) { - throw new AuthorizerCreationException(String.format("The %s '%s' is not a valid time interval.", propertyName, propertyValue)); - } - - return Math.toIntExact(timeoutValue); - } - - /** - * Called immediately before instance destruction for implementers to release resources. - * - * @throws AuthorizerDestructionException If pre-destruction fails. - */ - @Override - public void preDestruction() throws AuthorizerDestructionException { - try { - scheduler.shutdownNow(); - } catch (final Exception e) { - logger.warn("Error shutting down refresh scheduler", e); - } - try { - shellRunner.shutdown(); - } catch (final Exception e) { - logger.warn("Error shutting down ShellRunner", e); - } - } - - public ShellCommandsProvider getCommandsProvider() { - return selectedShellCommands; - } - - public void setCommandsProvider(ShellCommandsProvider commandsProvider) { - selectedShellCommands = commandsProvider; - } - - /** - * This is our entry point for user and group refresh. This method runs the top-level - * `getUserList()` and `getGroupsList()` shell commands, then passes those results to the - * other methods for record parse, extract, and object construction. - */ - private void refreshUsersAndGroups() { - final long startTime = System.currentTimeMillis(); - - Map<String, User> uidToUser = new HashMap<>(); - Map<String, User> usernameToUser = new HashMap<>(); - Map<String, User> gidToUser = new HashMap<>(); - Map<String, Group> gidToGroup = new HashMap<>(); - - List<String> userLines; - List<String> groupLines; - - try { - userLines = shellRunner.runShell(selectedShellCommands.getUsersList(), "Get Users List"); - groupLines = shellRunner.runShell(selectedShellCommands.getGroupsList(), "Get Groups List"); - } catch (final IOException ioexc) { - logger.error("refreshUsersAndGroups shell exception", ioexc); - return; - } - - rebuildUsers(userLines, uidToUser, usernameToUser, gidToUser); - rebuildGroups(groupLines, gidToGroup); - reconcilePrimaryGroups(gidToUser, gidToGroup); - - synchronized (usersById) { - usersById.clear(); - usersById.putAll(uidToUser); - - if (logger.isTraceEnabled()) { - logger.trace("=== Users by id..."); - Set<User> sortedUsers = new TreeSet<>(Comparator.comparing(User::getIdentity)); - sortedUsers.addAll(usersById.values()); - sortedUsers.forEach(u -> logger.trace("=== {}", u)); - } - } - - synchronized (usersByName) { - usersByName.clear(); - usersByName.putAll(usernameToUser); - logger.debug("users now size: {}", usersByName.size()); - } - - synchronized (groupsById) { - groupsById.clear(); - groupsById.putAll(gidToGroup); - logger.debug("groupsById now size: {}", groupsById.size()); - - if (logger.isTraceEnabled()) { - logger.trace("=== Groups by id..."); - Set<Group> sortedGroups = new TreeSet<>(Comparator.comparing(Group::getName)); - sortedGroups.addAll(groupsById.values()); - sortedGroups.forEach(g -> logger.trace("=== {}", g)); - } - } - - synchronized (groupsByName) { - groupsByName.clear(); - gidToGroup.values().forEach(g -> groupsByName.put(g.getName(), g)); - logger.debug("groupsByName now size: {}", groupsByName.size()); - } - - final long endTime = System.currentTimeMillis(); - logger.info("Refreshed users and groups, took {} seconds", (endTime - startTime) / 1000); - } - - /** - * This method parses the output of the `getUsersList()` shell command, where we expect the output - * to look like `user-name:user-id:primary-group-id`. - * <p> - * This method splits each output line on the ":" and attempts to build a User object - * from the resulting name, uid, and primary gid. Unusable records are logged. - */ - private void rebuildUsers(List<String> userLines, Map<String, User> idToUser, Map<String, User> usernameToUser, Map<String, User> gidToUser) { - userLines.forEach(line -> { - logger.trace("Processing user: {}", line); - - String[] record = line.split(":"); - if (record.length > 2) { - String userIdentity = record[0], userIdentifier = record[1], primaryGroupIdentifier = record[2]; - - if (!StringUtils.isBlank(userIdentifier) && !StringUtils.isBlank(userIdentity) && !excludeUsers.matcher(userIdentity).matches()) { - User user = new User.Builder() - .identity(userIdentity) - .identifierGenerateFromSeed(getUserIdentifierSeed(userIdentity)) - .build(); - idToUser.put(user.getIdentifier(), user); - usernameToUser.put(userIdentity, user); - logger.debug("Refreshed user {}", user); - - if (!StringUtils.isBlank(primaryGroupIdentifier)) { - // create a temporary group to deterministically generate the group id and associate this user - Group group = new Group.Builder() - .name(primaryGroupIdentifier) - .identifierGenerateFromSeed(getGroupIdentifierSeed(primaryGroupIdentifier)) - .build(); - gidToUser.put(group.getIdentifier(), user); - logger.debug("Associated primary group {} with user {}", group.getIdentifier(), userIdentity); - } else { - logger.warn("Null or empty primary group id for: {}", userIdentity); - } - - } else { - logger.warn("Null, empty, or skipped user name: {} or id: {}", userIdentity, userIdentifier); - } - } else { - logger.warn("Unexpected record format. Expected 3 or more colon separated values per line."); - } - }); - } - - /** - * This method parses the output of the `getGroupsList()` shell command, where we expect the output - * to look like `group-name:group-id`. - * <p> - * This method splits each output line on the ":" and attempts to build a Group object - * from the resulting name and gid. Unusable records are logged. - * <p> - * This command also runs the `getGroupMembers(username)` command once per group. The expected output - * of that command should look like `group-name-1,group-name-2`. - */ - private void rebuildGroups(List<String> groupLines, Map<String, Group> groupsById) { - groupLines.forEach(line -> { - logger.trace("Processing group: {}", line); - - String[] record = line.split(":"); - if (record.length > 1) { - Set<String> users = new HashSet<>(); - String groupName = record[0], groupIdentifier = record[1]; - - try { - String groupMembersCommand = selectedShellCommands.getGroupMembers(groupName); - List<String> memberLines = shellRunner.runShell(groupMembersCommand, "Get Group Members"); - // Use the first line only, and log if the line count isn't exactly one: - if (!memberLines.isEmpty()) { - String memberLine = memberLines.get(0); - if (!StringUtils.isBlank(memberLine)) { - String[] members = memberLine.split(","); - for (String userIdentity : members) { - if (!StringUtils.isBlank(userIdentity)) { - User tempUser = new User.Builder() - .identity(userIdentity) - .identifierGenerateFromSeed(getUserIdentifierSeed(userIdentity)) - .build(); - users.add(tempUser.getIdentifier()); - logger.debug("Added temp user {} for group {}", tempUser, groupName); - } - } - } else { - logger.debug("list membership returned no members"); - } - } else { - logger.debug("list membership returned zero lines."); - } - if (memberLines.size() > 1) { - logger.error("list membership returned too many lines, only used the first."); - } - - } catch (final IOException ioexc) { - logger.error("list membership shell exception", ioexc); - } - - if (!StringUtils.isBlank(groupIdentifier) && !StringUtils.isBlank(groupName) && !excludeGroups.matcher(groupName).matches()) { - Group group = new Group.Builder() - .name(groupName) - .identifierGenerateFromSeed(getGroupIdentifierSeed(groupIdentifier)) - .addUsers(users) - .build(); - groupsById.put(group.getIdentifier(), group); - logger.debug("Refreshed group {}", group); - } else { - logger.warn("Null, empty, or skipped group name: {} or id: {}", groupName, groupIdentifier); - } - } else { - logger.warn("Unexpected record format. Expected 1 or more comma separated values."); - } - }); - } - - /** - * This method parses the output of the `getGroupsList()` shell command, where we expect the output - * to look like `group-name:group-id`. - * <p> - * This method splits each output line on the ":" and attempts to build a Group object - * from the resulting name and gid. - */ - private void reconcilePrimaryGroups(Map<String, User> uidToUser, Map<String, Group> gidToGroup) { - uidToUser.forEach((primaryGid, primaryUser) -> { - Group primaryGroup = gidToGroup.get(primaryGid); - - if (primaryGroup == null) { - logger.warn("Primary group {} not found for {}", primaryGid, primaryUser.getIdentity()); - } else if (!excludeGroups.matcher(primaryGroup.getName()).matches()) { - Set<String> groupUsers = primaryGroup.getUsers(); - if (!groupUsers.contains(primaryUser.getIdentifier())) { - Set<String> updatedUserIdentifiers = new HashSet<>(groupUsers); - updatedUserIdentifiers.add(primaryUser.getIdentifier()); - - Group updatedGroup = new Group.Builder() - .identifier(primaryGroup.getIdentifier()) - .name(primaryGroup.getName()) - .addUsers(updatedUserIdentifiers) - .build(); - gidToGroup.put(updatedGroup.getIdentifier(), updatedGroup); - logger.debug("Added user {} to primary group {}", primaryUser, updatedGroup); - } else { - logger.debug("Primary group {} already contains user {}", primaryGroup, primaryUser); - } - } else { - logger.debug("Primary group {} excluded from matcher for {}", primaryGroup.getName(), primaryUser.getIdentity()); - } - }); - } - - private String getUserIdentifierSeed(final String userIdentifier) { - return legacyIdentifierMode ? userIdentifier : userIdentifier + "-user"; - } - - private String getGroupIdentifierSeed(final String groupIdentifier) { - return legacyIdentifierMode ? groupIdentifier : groupIdentifier + "-group"; - } - - - /** - * @return The fixed refresh delay. - */ - public long getRefreshDelay() { - return fixedDelay; - } - - /** - * Testing concession for clearing the internal caches. - */ - void clearCaches() { - synchronized (usersById) { - usersById.clear(); - } - - synchronized (usersByName) { - usersByName.clear(); - } - - synchronized (groupsById) { - groupsById.clear(); - } - - synchronized (groupsByName) { - groupsByName.clear(); - } - } - - /** - * @return The size of the internal user cache. - */ - public int userCacheSize() { - return usersById.size(); - } - - /** - * @return The size of the internal group cache. - */ - public int groupCacheSize() { - return groupsById.size(); - } -} diff --git a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/util/ShellRunner.java b/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/util/ShellRunner.java deleted file mode 100644 index fa1eceedf4..0000000000 --- a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/java/org/apache/nifi/authorization/util/ShellRunner.java +++ /dev/null @@ -1,123 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.authorization.util; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStreamReader; -import java.io.Reader; -import java.util.ArrayList; -import java.util.List; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.concurrent.ThreadFactory; -import java.util.concurrent.TimeUnit; - -public class ShellRunner { - private final static Logger logger = LoggerFactory.getLogger(ShellRunner.class); - - static String SHELL = "sh"; - static String OPTS = "-c"; - - private final int timeoutSeconds; - private final ExecutorService executor; - - public ShellRunner(final int timeoutSeconds) { - this.timeoutSeconds = timeoutSeconds; - this.executor = Executors.newFixedThreadPool(1, new ThreadFactory() { - @Override - public Thread newThread(final Runnable r) { - final Thread t = Executors.defaultThreadFactory().newThread(r); - t.setName("ShellRunner"); - t.setDaemon(true); - return t; - } - }); - } - - public List<String> runShell(String command, String description) throws IOException { - final ProcessBuilder builder = new ProcessBuilder(SHELL, OPTS, command); - builder.redirectErrorStream(true); - - final List<String> builderCommand = builder.command(); - logger.debug("Run Command '{}': {}", description, builderCommand); - - final Process proc = builder.start(); - - final List<String> lines = new ArrayList<>(); - executor.submit(() -> { - try { - try (final Reader stdin = new InputStreamReader(proc.getInputStream()); - final BufferedReader reader = new BufferedReader(stdin)) { - logger.trace("Reading process input stream..."); - - String line; - int lineCount = 0; - while ((line = reader.readLine()) != null) { - if (logger.isTraceEnabled()) { - logger.trace("{} - {}", (++lineCount), line); - } - lines.add(line.trim()); - } - - logger.trace("Finished reading process input stream"); - } - } catch (IOException e) { - logger.error(e.getMessage(), e); - } - }); - - boolean completed; - try { - completed = proc.waitFor(timeoutSeconds, TimeUnit.SECONDS); - } catch (InterruptedException irexc) { - throw new IOException(irexc.getMessage(), irexc.getCause()); - } - - if (!completed) { - logger.debug("Process did not complete in allotted time, attempting to forcibly destroy process..."); - try { - proc.destroyForcibly(); - } catch (Exception e) { - logger.debug("Process failed to destroy", e); - } - throw new IllegalStateException("Shell command '" + command + "' did not complete during the allotted time period"); - } - - if (proc.exitValue() != 0) { - throw new IOException("Process exited with non-zero value: " + proc.exitValue()); - } - - return lines; - } - - public void shutdown() { - executor.shutdown(); - try { - if (!executor.awaitTermination(5000L, TimeUnit.MILLISECONDS)) { - logger.info("Failed to stop ShellRunner executor in 5 seconds. Terminating"); - executor.shutdownNow(); - } - } catch (InterruptedException ie) { - executor.shutdownNow(); - Thread.currentThread().interrupt(); - } - } -} diff --git a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/resources/META-INF/services/org.apache.nifi.authorization.UserGroupProvider b/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/resources/META-INF/services/org.apache.nifi.authorization.UserGroupProvider deleted file mode 100755 index fb3360ddb8..0000000000 --- a/nifi-framework-bundle/nifi-framework/nifi-shell-authorizer/src/main/resources/META-INF/services/org.apache.nifi.authorization.UserGroupProvider +++ /dev/null @@ -1,15 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -org.apache.nifi.authorization.ShellUserGroupProvider diff --git a/nifi-framework-bundle/nifi-framework/pom.xml b/nifi-framework-bundle/nifi-framework/pom.xml index 54b62d6cac..e51c11ead3 100644 --- a/nifi-framework-bundle/nifi-framework/pom.xml +++ b/nifi-framework-bundle/nifi-framework/pom.xml @@ -54,7 +54,6 @@ <module>nifi-properties-loader</module> <module>nifi-standard-prioritizers</module> <module>nifi-mock-authorizer</module> - <module>nifi-shell-authorizer</module> <module>nifi-headless-server</module> </modules> </project> diff --git a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/NssShellCommands.java b/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/NssShellCommands.java deleted file mode 100644 index 0d49a6e0a0..0000000000 --- a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/NssShellCommands.java +++ /dev/null @@ -1,55 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.registry.security.authorization.shell; - -/** - * Provides shell commands to read users and groups on NSS-enabled systems. - * - * See `man 5 nsswitch.conf` for more info. - */ -class NssShellCommands implements ShellCommandsProvider { - /** - * @return Shell command string that will return a list of users. - */ - public String getUsersList() { - return "getent passwd | cut -f 1,3,4 -d ':'"; - } - - /** - * @return Shell command string that will return a list of groups. - */ - public String getGroupsList() { - return "getent group | cut -f 1,3 -d ':'"; - } - - /** - * @param groupName name of group. - * @return Shell command string that will return a list of users for a group. - */ - public String getGroupMembers(String groupName) { - return String.format("getent group %s | cut -f 4 -d ':'", groupName); - } - - /** - * This gives exit code 0 on all tested distributions. - * - * @return Shell command string that will exit normally (0) on a suitable system. - */ - public String getSystemCheck() { - return "getent --version"; - } -} \ No newline at end of file diff --git a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/OsxShellCommands.java b/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/OsxShellCommands.java deleted file mode 100644 index e4949ac2e5..0000000000 --- a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/OsxShellCommands.java +++ /dev/null @@ -1,55 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.registry.security.authorization.shell; - -/** - * Provides shell commands to read users and groups on Mac OSX systems. - * - * See `man dscl` for more info. - */ -class OsxShellCommands implements ShellCommandsProvider { - /** - * @return Shell command string that will return a list of users. - */ - public String getUsersList() { - return "dscl . -readall /Users UniqueID PrimaryGroupID | awk 'BEGIN { OFS = \":\"; ORS=\"\\n\"; i=0;} /RecordName: / {name = $2;i = 0;}" + - "/PrimaryGroupID: / {gid = $2;} /^ / {if (i == 0) { i++; name = $1;}} /UniqueID: / {uid = $2;print name, uid, gid;}' | grep -v ^_"; - } - - /** - * @return Shell command string that will return a list of groups. - */ - public String getGroupsList() { - return "dscl . -list /Groups PrimaryGroupID | grep -v '^_' | sed 's/ \\{1,\\}/:/g'"; - } - - /** - * - * @param groupName name of group. - * @return Shell command string that will return a list of users for a group. - */ - public String getGroupMembers(String groupName) { - return String.format("dscl . -read /Groups/%s GroupMembership | cut -f 2- -d ' ' | sed 's/\\ /,/g'", groupName); - } - - /** - * @return Shell command string that will exit normally (0) on a suitable system. - */ - public String getSystemCheck() { - return "which dscl"; - } -} diff --git a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/ShellCommandsProvider.java b/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/ShellCommandsProvider.java deleted file mode 100644 index 1038358637..0000000000 --- a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/ShellCommandsProvider.java +++ /dev/null @@ -1,67 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.registry.security.authorization.shell; - -/** - * Common interface for shell command strings to read users and groups. - * - */ -interface ShellCommandsProvider { - /** - * Gets the command for listing users. - * - * When executed, this command should output one record per line in this format: - * - * `username:user-id:primary-group-id` - * - * @return Shell command string that will return a list of users. - */ - String getUsersList(); - - /** - * Gets the command for listing groups. - * - * When executed, this command should output one record per line in this format: - * - * `group-name:group-id` - * - * @return Shell command string that will return a list of groups. - */ - String getGroupsList(); - - /** - * Gets the command for listing the members of a group. - * - * When executed, this command should output one line in this format: - * - * `user-name-1,user-name-2,user-name-n` - * - * @param groupName name of group. - * @return Shell command string that will return a list of users for a group. - */ - String getGroupMembers(String groupName); - - /** - * Gets the command for checking the suitability of the host system. - * - * The command is expected to exit with status 0 (zero) to indicate success, and any other status - * to indicate failure. - * - * @return Shell command string that will exit normally (0) on a suitable system. - */ - String getSystemCheck(); -} diff --git a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/ShellRunner.java b/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/ShellRunner.java deleted file mode 100644 index 8ad5b9ceb6..0000000000 --- a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/ShellRunner.java +++ /dev/null @@ -1,123 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.registry.security.authorization.shell; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStreamReader; -import java.io.Reader; -import java.util.ArrayList; -import java.util.List; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.concurrent.ThreadFactory; -import java.util.concurrent.TimeUnit; - -public class ShellRunner { - private final static Logger logger = LoggerFactory.getLogger(ShellRunner.class); - - static String SHELL = "sh"; - static String OPTS = "-c"; - - private final int timeoutSeconds; - private final ExecutorService executor; - - public ShellRunner(final int timeoutSeconds) { - this.timeoutSeconds = timeoutSeconds; - this.executor = Executors.newFixedThreadPool(1, new ThreadFactory() { - @Override - public Thread newThread(final Runnable r) { - final Thread t = Executors.defaultThreadFactory().newThread(r); - t.setName("ShellRunner"); - t.setDaemon(true); - return t; - } - }); - } - - public List<String> runShell(String command, String description) throws IOException { - final ProcessBuilder builder = new ProcessBuilder(SHELL, OPTS, command); - builder.redirectErrorStream(true); - - final List<String> builderCommand = builder.command(); - logger.debug("Run Command '{}': {}", description, builderCommand); - - final Process proc = builder.start(); - - final List<String> lines = new ArrayList<>(); - executor.submit(() -> { - try { - try (final Reader stdin = new InputStreamReader(proc.getInputStream()); - final BufferedReader reader = new BufferedReader(stdin)) { - logger.trace("Reading process input stream..."); - - String line; - int lineCount = 0; - while ((line = reader.readLine()) != null) { - if (logger.isTraceEnabled()) { - logger.trace("{} - {}", (++lineCount), line); - } - lines.add(line.trim()); - } - - logger.trace("Finished reading process input stream"); - } - } catch (IOException e) { - logger.error(e.getMessage(), e); - } - }); - - boolean completed; - try { - completed = proc.waitFor(timeoutSeconds, TimeUnit.SECONDS); - } catch (InterruptedException irexc) { - throw new IOException(irexc.getMessage(), irexc.getCause()); - } - - if (!completed) { - logger.debug("Process did not complete in allotted time, attempting to forcibly destroy process..."); - try { - proc.destroyForcibly(); - } catch (Exception e) { - logger.debug("Process failed to destroy", e); - } - throw new IllegalStateException("Shell command '" + command + "' did not complete during the allotted time period"); - } - - if (proc.exitValue() != 0) { - throw new IOException("Process exited with non-zero value: " + proc.exitValue()); - } - - return lines; - } - - public void shutdown() { - executor.shutdown(); - try { - if (!executor.awaitTermination(5000L, TimeUnit.MILLISECONDS)) { - logger.info("Failed to stop ShellRunner executor in 5 seconds. Terminating"); - executor.shutdownNow(); - } - } catch (InterruptedException ie) { - executor.shutdownNow(); - Thread.currentThread().interrupt(); - } - } -} diff --git a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/ShellUserGroupProvider.java b/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/ShellUserGroupProvider.java deleted file mode 100644 index 13f32f516c..0000000000 --- a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/security/authorization/shell/ShellUserGroupProvider.java +++ /dev/null @@ -1,645 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.nifi.registry.security.authorization.shell; - -import org.apache.commons.lang3.StringUtils; -import org.apache.nifi.registry.security.authorization.AuthorizerConfigurationContext; -import org.apache.nifi.registry.security.authorization.Group; -import org.apache.nifi.registry.security.authorization.User; -import org.apache.nifi.registry.security.authorization.UserAndGroups; -import org.apache.nifi.registry.security.authorization.UserGroupProvider; -import org.apache.nifi.registry.security.authorization.UserGroupProviderInitializationContext; -import org.apache.nifi.registry.security.authorization.exception.AuthorizationAccessException; -import org.apache.nifi.registry.security.exception.SecurityProviderCreationException; -import org.apache.nifi.registry.security.exception.SecurityProviderDestructionException; -import org.apache.nifi.registry.util.FormatUtils; -import org.apache.nifi.registry.util.PropertyValue; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.IOException; -import java.util.Comparator; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.TreeSet; -import java.util.concurrent.Executors; -import java.util.concurrent.ScheduledExecutorService; -import java.util.concurrent.TimeUnit; -import java.util.regex.Pattern; -import java.util.regex.PatternSyntaxException; - -/* - * ShellUserGroupProvider implements UserGroupProvider by way of shell commands. - */ -public class ShellUserGroupProvider implements UserGroupProvider { - private final static Logger logger = LoggerFactory.getLogger(ShellUserGroupProvider.class); - - private final static String OS_TYPE_ERROR = "Unsupported operating system."; - private final static String SYS_CHECK_ERROR = "System check failed - cannot provide users and groups."; - private final static Map<String, User> usersById = new HashMap<>(); // id == identifier - private final static Map<String, User> usersByName = new HashMap<>(); // name == identity - private final static Map<String, Group> groupsById = new HashMap<>(); - - public static final String REFRESH_DELAY_PROPERTY = "Refresh Delay"; - private static final long MINIMUM_SYNC_INTERVAL_MILLISECONDS = 10_000; - - public static final String EXCLUDE_USER_PROPERTY = "Exclude Users"; - public static final String EXCLUDE_GROUP_PROPERTY = "Exclude Groups"; - - public static final String COMMAND_TIMEOUT_PROPERTY = "Command Timeout"; - - private static final String DEFAULT_COMMAND_TIMEOUT = "60 seconds"; - - private long fixedDelay; - private Pattern excludeUsers; - private Pattern excludeGroups; - private int timeoutSeconds; - - // Our scheduler has one thread for users, one for groups: - private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(2); - - // Commands selected during initialization: - private ShellCommandsProvider selectedShellCommands; - - private ShellRunner shellRunner; - - // Start of the UserGroupProvider implementation. Javadoc strings - // copied from the interface definition for reference. - - /** - * Retrieves all users. Must be non null - * - * @return a list of users - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public Set<User> getUsers() throws AuthorizationAccessException { - synchronized (usersById) { - logger.debug("getUsers has user set of size: {}", usersById.size()); - return new HashSet<>(usersById.values()); - } - } - - /** - * Retrieves the user with the given identifier. - * - * @param identifier the id of the user to retrieve - * @return the user with the given id, or null if no matching user was found - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public User getUser(String identifier) throws AuthorizationAccessException { - User user; - - synchronized (usersById) { - user = usersById.get(identifier); - } - - if (user == null) { - logger.debug("getUser (by id) user not found: {}", identifier); - } else { - logger.debug("getUser (by id) found user: {} for id: {}", user, identifier); - } - return user; - } - - /** - * Retrieves the user with the given identity. - * - * @param identity the identity of the user to retrieve - * @return the user with the given identity, or null if no matching user was found - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public User getUserByIdentity(String identity) throws AuthorizationAccessException { - User user; - - synchronized (usersByName) { - user = usersByName.get(identity); - } - - if (user == null) { - logger.debug("getUser (by name) user not found: {}", identity); - } else { - logger.debug("getUser (by name) found user: {} for name: {}", user.getIdentity(), identity); - } - return user; - } - - /** - * Retrieves all groups. Must be non null - * - * @return a list of groups - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public Set<Group> getGroups() throws AuthorizationAccessException { - synchronized (groupsById) { - logger.debug("getGroups has group set of size: {}", groupsById.size()); - return new HashSet<>(groupsById.values()); - } - } - - /** - * Retrieves a Group by Id. - * - * @param identifier the identifier of the Group to retrieve - * @return the Group with the given identifier, or null if no matching group was found - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public Group getGroup(String identifier) throws AuthorizationAccessException { - Group group; - - synchronized (groupsById) { - group = groupsById.get(identifier); - } - - if (group == null) { - logger.debug("getGroup (by id) group not found: {}", identifier); - } else { - logger.debug("getGroup (by id) found group: {} for id: {}", group.getName(), identifier); - } - return group; - - } - - /** - * Gets a user and their groups. - * - * @return the UserAndGroups for the specified identity - * @throws AuthorizationAccessException if there was an unexpected error performing the operation - */ - @Override - public UserAndGroups getUserAndGroups(String identity) throws AuthorizationAccessException { - User user = getUserByIdentity(identity); - logger.debug("Retrieved user {} for identity {}", user, identity); - - Set<Group> groups = new HashSet<>(); - if (user != null) { - for (Group g : getGroups()) { - if (g.getUsers().contains(user.getIdentifier())) { - logger.debug("User {} belongs to group {}", user.getIdentity(), g.getName()); - groups.add(g); - } - } - } - - if (groups.isEmpty()) { - logger.debug("User {} belongs to no groups", user); - } - - return new UserAndGroups() { - @Override - public User getUser() { - return user; - } - - @Override - public Set<Group> getGroups() { - return groups; - } - }; - } - - /** - * Called immediately after instance creation for implementers to perform additional setup - * - * @param initializationContext in which to initialize - */ - @Override - public void initialize(UserGroupProviderInitializationContext initializationContext) throws SecurityProviderCreationException { - } - - /** - * Called to configure the Authorizer. - * - * @param configurationContext at the time of configuration - * @throws SecurityProviderCreationException for any issues configuring the provider - */ - @Override - public void onConfigured(AuthorizerConfigurationContext configurationContext) throws SecurityProviderCreationException { - logger.info("Configuring ShellUserGroupProvider"); - - fixedDelay = getDelayProperty(configurationContext, REFRESH_DELAY_PROPERTY, "5 mins"); - timeoutSeconds = getTimeoutProperty(configurationContext, COMMAND_TIMEOUT_PROPERTY, DEFAULT_COMMAND_TIMEOUT); - shellRunner = new ShellRunner(timeoutSeconds); - logger.debug("Configured ShellRunner with command timeout of '{}' seconds", timeoutSeconds); - - - // Our next init step is to select the command set based on the operating system name: - ShellCommandsProvider commands = getCommandsProvider(); - - if (commands == null) { - commands = getCommandsProviderFromName(null); - setCommandsProvider(commands); - } - - // Our next init step is to run the system check from that command set to determine if the other commands - // will work on this host or not. - try { - shellRunner.runShell(commands.getSystemCheck(), "Supported System Check"); - } catch (final Exception e) { - logger.error("initialize exception: system check command: {}", commands.getSystemCheck(), e); - throw new SecurityProviderCreationException(SYS_CHECK_ERROR, e); - } - - // The next step is to add the user and group exclude regexes: - try { - excludeGroups = Pattern.compile(getProperty(configurationContext, EXCLUDE_GROUP_PROPERTY, "")); - excludeUsers = Pattern.compile(getProperty(configurationContext, EXCLUDE_USER_PROPERTY, "")); - } catch (final PatternSyntaxException e) { - throw new SecurityProviderCreationException(e); - } - - // With our command set selected, and our system check passed, we can pull in the users and groups: - refreshUsersAndGroups(); - - // And finally, our last init step is to fire off the refresh thread: - scheduler.scheduleWithFixedDelay(() -> { - try { - refreshUsersAndGroups(); - } catch (final Throwable t) { - logger.error("", t); - } - }, fixedDelay, fixedDelay, TimeUnit.MILLISECONDS); - - logger.info("Completed configuration of ShellUserGroupProvider"); - } - - private static ShellCommandsProvider getCommandsProviderFromName(String osName) { - if (osName == null) { - osName = System.getProperty("os.name"); - } - - ShellCommandsProvider commands; - if (osName.startsWith("Linux")) { - logger.debug("Selected Linux command set."); - commands = new NssShellCommands(); - } else if (osName.startsWith("Mac OS X")) { - logger.debug("Selected OSX command set."); - commands = new OsxShellCommands(); - } else { - throw new SecurityProviderCreationException(OS_TYPE_ERROR); - } - return commands; - } - - private String getProperty(AuthorizerConfigurationContext authContext, String propertyName, String defaultValue) { - final PropertyValue property = authContext.getProperty(propertyName); - final String value; - - if (property != null && property.isSet()) { - value = property.getValue(); - } else { - value = defaultValue; - } - return value; - - } - - private long getDelayProperty(AuthorizerConfigurationContext authContext, String propertyName, String defaultValue) { - final PropertyValue intervalProperty = authContext.getProperty(propertyName); - final String propertyValue; - final long syncInterval; - - if (intervalProperty.isSet()) { - propertyValue = intervalProperty.getValue(); - } else { - propertyValue = defaultValue; - } - - try { - syncInterval = Math.round(FormatUtils.getPreciseTimeDuration(propertyValue, TimeUnit.MILLISECONDS)); - } catch (final IllegalArgumentException ignored) { - throw new SecurityProviderCreationException(String.format("The %s '%s' is not a valid time interval.", propertyName, propertyValue)); - } - - if (syncInterval < MINIMUM_SYNC_INTERVAL_MILLISECONDS) { - throw new SecurityProviderCreationException(String.format("The %s '%s' is below the minimum value of '%d ms'", propertyName, propertyValue, MINIMUM_SYNC_INTERVAL_MILLISECONDS)); - } - return syncInterval; - } - - private int getTimeoutProperty(AuthorizerConfigurationContext authContext, String propertyName, String defaultValue) { - final PropertyValue timeoutProperty = authContext.getProperty(propertyName); - - final String propertyValue; - if (timeoutProperty.isSet()) { - propertyValue = timeoutProperty.getValue(); - } else { - propertyValue = defaultValue; - } - - final long timeoutValue; - try { - timeoutValue = Math.round(FormatUtils.getPreciseTimeDuration(propertyValue, TimeUnit.SECONDS)); - } catch (final IllegalArgumentException ignored) { - throw new SecurityProviderCreationException(String.format("The %s '%s' is not a valid time interval.", propertyName, propertyValue)); - } - - return Math.toIntExact(timeoutValue); - } - - /** - * Called immediately before instance destruction for implementers to release resources. - * - * @throws SecurityProviderDestructionException If pre-destruction fails. - */ - @Override - public void preDestruction() throws SecurityProviderDestructionException { - try { - scheduler.shutdownNow(); - } catch (final Exception e) { - logger.warn("Error shutting down refresh scheduler", e); - } - try { - shellRunner.shutdown(); - } catch (final Exception e) { - logger.warn("Error shutting down ShellRunner", e); - } - } - - public ShellCommandsProvider getCommandsProvider() { - return selectedShellCommands; - } - - public void setCommandsProvider(ShellCommandsProvider commandsProvider) { - selectedShellCommands = commandsProvider; - } - - /** - * This is our entry point for user and group refresh. This method runs the top-level - * `getUserList()` and `getGroupsList()` shell commands, then passes those results to the - * other methods for record parse, extract, and object construction. - */ - private void refreshUsersAndGroups() { - final long startTime = System.currentTimeMillis(); - - Map<String, User> uidToUser = new HashMap<>(); - Map<String, User> usernameToUser = new HashMap<>(); - Map<String, User> gidToUser = new HashMap<>(); - Map<String, Group> gidToGroup = new HashMap<>(); - - List<String> userLines; - List<String> groupLines; - - try { - userLines = shellRunner.runShell(selectedShellCommands.getUsersList(), "Get Users List"); - groupLines = shellRunner.runShell(selectedShellCommands.getGroupsList(), "Get Groups List"); - } catch (final IOException ioexc) { - logger.error("refreshUsersAndGroups shell exception", ioexc); - return; - } - - rebuildUsers(userLines, uidToUser, usernameToUser, gidToUser); - rebuildGroups(groupLines, gidToGroup); - reconcilePrimaryGroups(gidToUser, gidToGroup); - - synchronized (usersById) { - usersById.clear(); - usersById.putAll(uidToUser); - - if (logger.isTraceEnabled()) { - logger.trace("=== Users by id..."); - Set<User> sortedUsers = new TreeSet<>(Comparator.comparing(User::getIdentity)); - sortedUsers.addAll(usersById.values()); - sortedUsers.forEach(u -> logger.trace("=== {}", u)); - } - } - - synchronized (usersByName) { - usersByName.clear(); - usersByName.putAll(usernameToUser); - logger.debug("users now size: {}", usersByName.size()); - } - - synchronized (groupsById) { - groupsById.clear(); - groupsById.putAll(gidToGroup); - logger.debug("groups now size: {}", groupsById.size()); - - if (logger.isTraceEnabled()) { - logger.trace("=== Groups by id..."); - Set<Group> sortedGroups = new TreeSet<>(Comparator.comparing(Group::getName)); - sortedGroups.addAll(groupsById.values()); - sortedGroups.forEach(g -> logger.trace("=== {}", g)); - } - } - - final long endTime = System.currentTimeMillis(); - logger.info("Refreshed users and groups, took {} seconds", (endTime - startTime) / 1000); - } - - /** - * This method parses the output of the `getUsersList()` shell command, where we expect the output - * to look like `user-name:user-id:primary-group-id`. - * <p> - * This method splits each output line on the ":" and attempts to build a User object - * from the resulting name, uid, and primary gid. Unusable records are logged. - */ - private void rebuildUsers(List<String> userLines, Map<String, User> idToUser, Map<String, User> usernameToUser, Map<String, User> gidToUser) { - userLines.forEach(line -> { - logger.trace("Processing user: {}", line); - - String[] record = line.split(":"); - if (record.length > 2) { - String userIdentity = record[0], userIdentifier = record[1], primaryGroupIdentifier = record[2]; - - if (!StringUtils.isBlank(userIdentifier) && !StringUtils.isBlank(userIdentity) && !excludeUsers.matcher(userIdentity).matches()) { - User user = new User.Builder() - .identity(userIdentity) - .identifierGenerateFromSeed(getUserIdentifierSeed(userIdentity)) - .build(); - idToUser.put(user.getIdentifier(), user); - usernameToUser.put(userIdentity, user); - logger.debug("Refreshed user {}", user); - - if (!StringUtils.isBlank(primaryGroupIdentifier)) { - // create a temporary group to deterministically generate the group id and associate this user - Group group = new Group.Builder() - .name(primaryGroupIdentifier) - .identifierGenerateFromSeed(getGroupIdentifierSeed(primaryGroupIdentifier)) - .build(); - gidToUser.put(group.getIdentifier(), user); - logger.debug("Associated primary group {} with user {}", group.getIdentifier(), userIdentity); - } else { - logger.warn("Null or empty primary group id for: {}", userIdentity); - } - - } else { - logger.warn("Null, empty, or skipped user name: {} or id: {}", userIdentity, userIdentifier); - } - } else { - logger.warn("Unexpected record format. Expected 3 or more colon separated values per line."); - } - }); - } - - /** - * This method parses the output of the `getGroupsList()` shell command, where we expect the output - * to look like `group-name:group-id`. - * <p> - * This method splits each output line on the ":" and attempts to build a Group object - * from the resulting name and gid. Unusable records are logged. - * <p> - * This command also runs the `getGroupMembers(username)` command once per group. The expected output - * of that command should look like `group-name-1,group-name-2`. - */ - private void rebuildGroups(List<String> groupLines, Map<String, Group> groupsById) { - groupLines.forEach(line -> { - logger.trace("Processing group: {}", line); - - String[] record = line.split(":"); - if (record.length > 1) { - Set<String> users = new HashSet<>(); - String groupName = record[0], groupIdentifier = record[1]; - - try { - String groupMembersCommand = selectedShellCommands.getGroupMembers(groupName); - List<String> memberLines = shellRunner.runShell(groupMembersCommand, "Get Group Members"); - // Use the first line only, and log if the line count isn't exactly one: - if (!memberLines.isEmpty()) { - String memberLine = memberLines.get(0); - if (!StringUtils.isBlank(memberLine)) { - String[] members = memberLine.split(","); - for (String userIdentity : members) { - if (!StringUtils.isBlank(userIdentity)) { - User tempUser = new User.Builder() - .identity(userIdentity) - .identifierGenerateFromSeed(getUserIdentifierSeed(userIdentity)) - .build(); - users.add(tempUser.getIdentifier()); - logger.debug("Added temp user {} for group {}", tempUser, groupName); - } - } - } else { - logger.debug("list membership returned no members"); - } - } else { - logger.debug("list membership returned zero lines."); - } - if (memberLines.size() > 1) { - logger.error("list membership returned too many lines, only used the first."); - } - - } catch (final IOException ioexc) { - logger.error("list membership shell exception", ioexc); - } - - if (!StringUtils.isBlank(groupIdentifier) && !StringUtils.isBlank(groupName) && !excludeGroups.matcher(groupName).matches()) { - Group group = new Group.Builder() - .name(groupName) - .identifierGenerateFromSeed(getGroupIdentifierSeed(groupIdentifier)) - .addUsers(users) - .build(); - groupsById.put(group.getIdentifier(), group); - logger.debug("Refreshed group {}", group); - } else { - logger.warn("Null, empty, or skipped group name: {} or id: {}", groupName, groupIdentifier); - } - } else { - logger.warn("Unexpected record format. Expected 1 or more comma separated values."); - } - }); - } - - /** - * This method parses the output of the `getGroupsList()` shell command, where we expect the output - * to look like `group-name:group-id`. - * <p> - * This method splits each output line on the ":" and attempts to build a Group object - * from the resulting name and gid. - */ - private void reconcilePrimaryGroups(Map<String, User> uidToUser, Map<String, Group> gidToGroup) { - uidToUser.forEach((primaryGid, primaryUser) -> { - Group primaryGroup = gidToGroup.get(primaryGid); - - if (primaryGroup == null) { - logger.warn("Primary group {} not found for {}", primaryGid, primaryUser.getIdentity()); - } else if (!excludeGroups.matcher(primaryGroup.getName()).matches()) { - Set<String> groupUsers = primaryGroup.getUsers(); - if (!groupUsers.contains(primaryUser.getIdentifier())) { - Set<String> updatedUserIdentifiers = new HashSet<>(groupUsers); - updatedUserIdentifiers.add(primaryUser.getIdentifier()); - - Group updatedGroup = new Group.Builder() - .identifier(primaryGroup.getIdentifier()) - .name(primaryGroup.getName()) - .addUsers(updatedUserIdentifiers) - .build(); - gidToGroup.put(updatedGroup.getIdentifier(), updatedGroup); - logger.debug("Added user {} to primary group {}", primaryUser, updatedGroup); - } else { - logger.debug("Primary group {} already contains user {}", primaryGroup, primaryUser); - } - } else { - logger.debug("Primary group {} excluded from matcher for {}", primaryGroup.getName(), primaryUser.getIdentity()); - } - }); - } - - private String getUserIdentifierSeed(final String userIdentifier) { - return userIdentifier + "-user"; - } - - private String getGroupIdentifierSeed(final String groupIdentifier) { - return groupIdentifier + "-group"; - } - - - /** - * @return The fixed refresh delay. - */ - public long getRefreshDelay() { - return fixedDelay; - } - - /** - * Testing concession for clearing the internal caches. - */ - void clearCaches() { - synchronized (usersById) { - usersById.clear(); - } - - synchronized (usersByName) { - usersByName.clear(); - } - - synchronized (groupsById) { - groupsById.clear(); - } - } - - /** - * @return The size of the internal user cache. - */ - public int userCacheSize() { - return usersById.size(); - } - - /** - * @return The size of the internal group cache. - */ - public int groupCacheSize() { - return groupsById.size(); - } -} diff --git a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/resources/META-INF/services/org.apache.nifi.registry.security.authorization.UserGroupProvider b/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/resources/META-INF/services/org.apache.nifi.registry.security.authorization.UserGroupProvider index 73ec0cf54c..6343af9460 100644 --- a/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/resources/META-INF/services/org.apache.nifi.registry.security.authorization.UserGroupProvider +++ b/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/resources/META-INF/services/org.apache.nifi.registry.security.authorization.UserGroupProvider @@ -16,5 +16,4 @@ org.apache.nifi.registry.security.authorization.CompositeUserGroupProvider org.apache.nifi.registry.security.authorization.CompositeConfigurableUserGroupProvider org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider -org.apache.nifi.registry.security.authorization.database.DatabaseUserGroupProvider -org.apache.nifi.registry.security.authorization.shell.ShellUserGroupProvider \ No newline at end of file +org.apache.nifi.registry.security.authorization.database.DatabaseUserGroupProvider \ No newline at end of file diff --git a/nifi-registry/nifi-registry-core/nifi-registry-resources/src/main/resources/conf/authorizers.xml b/nifi-registry/nifi-registry-core/nifi-registry-resources/src/main/resources/conf/authorizers.xml index 9db7acad82..ff01c8f073 100644 --- a/nifi-registry/nifi-registry-core/nifi-registry-resources/src/main/resources/conf/authorizers.xml +++ b/nifi-registry/nifi-registry-core/nifi-registry-resources/src/main/resources/conf/authorizers.xml @@ -179,27 +179,6 @@ </userGroupProvider> To enable the ldap-user-group-provider remove 2 lines. This is 2 of 2. --> - <!-- - The ShellUserGroupProvider provides support for retrieving users and groups by way of shell commands - on systems that support `sh`. Implementations available for Linux and Mac OS, and are selected by the - provider based on the system property `os.name`. - - 'Refresh Delay' - duration to wait between subsequent refreshes. Default is '5 mins'. - 'Exclude Groups' - regular expression used to exclude groups. Default is '', which means no groups are excluded. - 'Exclude Users' - regular expression used to exclude users. Default is '', which means no users are excluded. - 'Command Timeout' - amount of time to wait while executing a command before timing out - --> - <!-- To enable the shell-user-group-provider remove 2 lines. This is 1 of 2. - <userGroupProvider> - <identifier>shell-user-group-provider</identifier> - <class>org.apache.nifi.registry.security.authorization.shell.ShellUserGroupProvider</class> - <property name="Refresh Delay">5 mins</property> - <property name="Exclude Groups"></property> - <property name="Exclude Users"></property> - <property name="Command Timeout">60 seconds</property> - </userGroupProvider> - To enable the shell-user-group-provider remove 2 lines. This is 2 of 2. --> - <!-- The CompositeUserGroupProvider will provide support for retrieving users and groups from multiple sources.