[
https://issues.apache.org/jira/browse/YARN-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shanyu zhao updated YARN-9834:
------------------------------
Description:
Yarn Secure Container allows separation of different user's local files and
container processes running on the same node manager. This depends on an out of
band service such as SSSD/Winbind to sync all domain users to local machine
that runs Yarn node manager. *Hadoop code only works with local users*.
Winbind/SSSD user sync has lots of overhead, especially for large corporations.
Also if running Yarn node manager inside Kubernetes cluster (meaning node
managers running inside Docker container), it doesn't make sense for each
Docker container to domain join with Active Directory and sync a whole copy of
domain users to the Docker container.
We need an optional light-weighted approach to enable Yarn Secure Container in
secure mode, as an alternative to AD domain join and SSSD/Winbind based
user-sync service.
Today, class LinuxContainerExecutor already supports running Yarn container
process as one designated local user in non-secure mode.
*We can add new configurations to Yarn, such that with LinuxContainerExecutor
we can pre-create a pool of local users on each Yarn node manager. At runtime,
Yarn node manager allocates a local user to run the container process, for the
domain user that submits the application*. When all containers of that user are
finished and all files belonging to that user are deleted, we can release the
allocation and allow other users to use the same local user to run their Yarn
containers.
Please look at attached design doc for more details.
was:
Yarn Secure Container allows separation of different user's local files and
container processes running on the same node manager. This depends on an out of
band service such as SSSD/Winbind to sync all domain users to local machine
that runs Yarn node manager. * Hadoop code only works with local users*.
Winbind/SSSD user sync has lots of overhead, especially for large corporations.
Also if running Yarn node manager inside Kubernetes cluster (meaning node
managers running inside Docker container), it doesn't make sense for each
Docker container to domain join with Active Directory and sync a whole copy of
domain users to the Docker container.
We need an optional light-weighted approach to enable Yarn Secure Container in
secure mode, as an alternative to AD domain join and SSSD/Winbind based
user-sync service.
Today, class LinuxContainerExecutor already supports running Yarn container
process as one designated local user in non-secure mode.
*We can add new configurations to Yarn, such that with LinuxContainerExecutor
we can pre-create a pool of local users on each Yarn node manager. At runtime,
Yarn node manager allocates a local user to run the container process, for the
domain user that submits the application*. When all containers of that user are
finished and all files belonging to that user are deleted, we can release the
allocation and allow other users to use the same local user to run their Yarn
containers.
Please look at attached design doc for more details.
> Allow using a pool of local users to run Yarn Secure Container in secure mode
> -----------------------------------------------------------------------------
>
> Key: YARN-9834
> URL: https://issues.apache.org/jira/browse/YARN-9834
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 3.1.2
> Reporter: shanyu zhao
> Assignee: shanyu zhao
> Priority: Major
>
> Yarn Secure Container allows separation of different user's local files and
> container processes running on the same node manager. This depends on an out
> of band service such as SSSD/Winbind to sync all domain users to local
> machine that runs Yarn node manager. *Hadoop code only works with local
> users*.
> Winbind/SSSD user sync has lots of overhead, especially for large
> corporations. Also if running Yarn node manager inside Kubernetes cluster
> (meaning node managers running inside Docker container), it doesn't make
> sense for each Docker container to domain join with Active Directory and sync
> a whole copy of domain users to the Docker container.
> We need an optional light-weighted approach to enable Yarn Secure Container
> in secure mode, as an alternative to AD domain join and SSSD/Winbind based
> user-sync service.
> Today, class LinuxContainerExecutor already supports running Yarn container
> process as one designated local user in non-secure mode.
> *We can add new configurations to Yarn, such that with LinuxContainerExecutor
> we can pre-create a pool of local users on each Yarn node manager. At
> runtime, Yarn node manager allocates a local user to run the container
> process, for the domain user that submits the application*. When all
> containers of that user are finished and all files belonging to that user are
> deleted, we can release the allocation and allow other users to use the same
> local user to run their Yarn containers.
> Please look at attached design doc for more details.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
