[
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942957#comment-16942957
]
Shane Kumpf commented on YARN-9860:
-----------------------------------
{quote}It looks like we are reverting to our old habit of using 0 for true. It
would be more consistent to use is_feature_enabled() method to determine if
service_mode is enabled, and reduce some code debris.
{quote}
Good points, I agree these comments should be addressed.
{quote}Container-executor already dup the container run output into stdout and
stderr log files with proper user permission for entrypoint mode because the
log files are initialized as user who runs the container executor rather than
the user in the container. It works for both secure and non-secure mode. I fail
to see the need to craft logging mechanism for the given reasoning for service
mode. Let me know if I missed something.
{quote}
Excellent, I can confirm this is working exactly how we'd want. I overlooked
this before. Seems logging isn't an issue after all. Thanks for pointing that
out!
I did retest the patch today and it is still working as expected.
With the patch applied in my dev VM, below is the ps and logs from the official
postgres image running under YARN with zero changes!
*ps:*
{code:java}
root@centos7-0:/# ps -ef
UID PID PPID C STIME TTY TIME CMD
postgres 1 0 0 16:13 ? 00:00:00 postgres
postgres 53 1 0 16:13 ? 00:00:00 postgres: checkpointer
postgres 54 1 0 16:13 ? 00:00:00 postgres: background writer
postgres 55 1 0 16:13 ? 00:00:00 postgres: walwriter
postgres 56 1 0 16:13 ? 00:00:00 postgres: autovacuum launcher
postgres 57 1 0 16:13 ? 00:00:00 postgres: stats collector
postgres 58 1 0 16:13 ? 00:00:00 postgres: logical replication
launcher
root 59 0 4 16:14 pts/0 00:00:00 bash
root 64 59 0 16:14 pts/0 00:00:00 ps -ef
{code}
*Logs:*
{code:java}
[root@y7001 ~]# yarn logs -applicationId application_1570018164872_0005
-containerId container_1570018164872_0005_01_000002
2019-10-02 16:26:40,269 INFO client.RMProxy: Connecting to ResourceManager at
y7001.yns.foo.com/192.168.70.211:9104
Container: container_1570018164872_0005_01_000002 on y7001.yns.foo.com:9105
LogAggregationType: LOCAL
===================================================================================
LogType:stdout.txt
LogLastModifiedTime:Wed Oct 02 16:13:31 +0000 2019
LogLength:2638
LogContents:
Launching docker container...
Docker run command: /usr/bin/docker run
--name=container_1570018164872_0005_01_000002 --net=host -v
/tmp/hadoop-yarn/nm-local-dir/filecache/13/httpd-proxy.conf:/etc/httpd/conf.d/httpd-proxy.conf:ro
--cgroup-parent=/hadoop-yarn/container_1570018164872_0005_01_000002
--cap-drop=ALL --cap-add=SYS_CHROOT --cap-add=MKNOD --cap-add=SETFCAP
--cap-add=SETPCAP --cap-add=FSETID --cap-add=CHOWN --cap-add=AUDIT_WRITE
--cap-add=SETGID --cap-add=NET_RAW --cap-add=FOWNER --cap-add=SETUID
--cap-add=DAC_OVERRIDE --cap-add=KILL --cap-add=NET_BIND_SERVICE
--hostname=centos7-0.skumpftest.hadoopuser.ynsdev --env-file
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1570018164872_0005/container_1570018164872_0005_01_000002/docker.container_1570018164872_0005_01_0000028354800474788286290.env
library/postgres
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....2019-10-02 16:13:31.231 UTC [43] LOG: listening
on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-10-02 16:13:31.253 UTC [44] LOG: database system was shut down at
2019-10-02 16:13:30 UTC
2019-10-02 16:13:31.259 UTC [43] LOG: database system is ready to accept
connections
done
server started
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
waiting for server to shut down...2019-10-02 16:13:31.322 UTC [43] LOG:
received fast shutdown request
.2019-10-02 16:13:31.325 UTC [43] LOG: aborting any active transactions
2019-10-02 16:13:31.329 UTC [43] LOG: background worker "logical replication
launcher" (PID 50) exited with exit code 1
2019-10-02 16:13:31.329 UTC [45] LOG: shutting down
2019-10-02 16:13:31.351 UTC [43] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
End of LogType:stdout.txt.This log file belongs to a running container
(container_1570018164872_0005_01_000002) and so may not be complete.
***************************************************************************
Container: container_1570018164872_0005_01_000002 on y7001.yns.foo.com:9105
LogAggregationType: LOCAL
===================================================================================
LogType:stderr.txt
LogLastModifiedTime:Wed Oct 02 16:13:31 +0000 2019
LogLength:3096
LogContents:
Unable to find image 'postgres:latest' locally
latest: Pulling from library/postgres
-snip-
2019-10-02 16:13:31.442 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port
5432
2019-10-02 16:13:31.443 UTC [1] LOG: listening on IPv6 address "::", port 5432
2019-10-02 16:13:31.446 UTC [1] LOG: listening on Unix socket
"/var/run/postgresql/.s.PGSQL.5432"
2019-10-02 16:13:31.463 UTC [52] LOG: database system was shut down at
2019-10-02 16:13:31 UTC
2019-10-02 16:13:31.468 UTC [1] LOG: database system is ready to accept
connections
End of LogType:stderr.txt.This log file belongs to a running container
(container_1570018164872_0005_01_000002) and so may not be complete.
***************************************************************************
{code}
> Enable service mode for Docker containers on YARN
> -------------------------------------------------
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 3.3.0
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service
> Mode".
> Service Mode - Run the container as defined by the image, but still allow for
> injecting configuration.
> Background:
> Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as
> defined in the image. However, still requires modification to official images
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>
> Implementation:
> Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
> Must be requested at runtime - (example:
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
> Entrypoint mode is default enabled for this mode (If Service Mode is
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set
> to true)
> Writable log mount will not be added - stdout logging may still work
> with entrypoint mode - remove the writable bind mounts
> User and groups will not be propagated (now: docker run --user nobody
> --group-add=nobody .... <image>, after: docker run .... <image>)
> Read-only resources mounted at the file level, files get chmod 777,
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
