[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841168#comment-17841168
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 merged PR #6771:
URL: https://github.com/apache/hadoop/pull/6771
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841167#comment-17841167
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2079194163
Thanks @tomicooler for the review, merging to trunk.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841158#comment-17841158
]
ASF GitHub Bot commented on YARN-11690:
---
hadoop-yetus commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2079174965
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 18s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 91m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed |
| +1 :green_heart: | cc | 1m 14s | | the patch passed |
| +1 :green_heart: | golang | 1m 14s | | the patch passed |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 46s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 162m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6771 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 26c72a68eecc 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ed587f807d2eb5e93978a0faf1c77b4e506c9f21 |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/4/testReport/ |
| Max. process+thread count | 534 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/4/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group =
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841100#comment-17841100
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 commented on code in PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#discussion_r1580673804
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h:
##
@@ -80,6 +80,11 @@ enum operations {
#define ROOT_VAR_TMP_DIR "private_var_slash_tmp"
#define COMMAND_FILE_SECTION "command-execution"
+// include cgroup2 super magic number if not defined
+#ifndef CGROUP2_SUPER_MAGIC
Review Comment:
Done.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841014#comment-17841014
]
ASF GitHub Bot commented on YARN-11690:
---
tomicooler commented on code in PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#discussion_r1580335818
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h:
##
@@ -80,6 +80,11 @@ enum operations {
#define ROOT_VAR_TMP_DIR "private_var_slash_tmp"
#define COMMAND_FILE_SECTION "command-execution"
+// include cgroup2 super magic number if not defined
+#ifndef CGROUP2_SUPER_MAGIC
Review Comment:
This must be done after the magic.h include. Let's move this to the .c file
somewhere after the include.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840867#comment-17840867
]
ASF GitHub Bot commented on YARN-11690:
---
hadoop-yetus commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2077695501
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 45m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 51m 7s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 91m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed |
| +1 :green_heart: | cc | 1m 13s | | the patch passed |
| +1 :green_heart: | golang | 1m 13s | | the patch passed |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 48s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 206m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6771 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 3dd51f4f1063 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1c9e812da5d2ca575563caa8c0b59351419ebb44 |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/3/testReport/ |
| Max. process+thread count | 532 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/3/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group =
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840819#comment-17840819
]
ASF GitHub Bot commented on YARN-11690:
---
hadoop-yetus commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2077302452
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 13s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 90m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed |
| +1 :green_heart: | cc | 1m 14s | | the patch passed |
| +1 :green_heart: | golang | 1m 14s | | the patch passed |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 29s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 161m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6771 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 32f47319a7b7 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 64256e8f1b3ce6fb9f9bf79c8b9c04a057f7b72c |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/2/testReport/ |
| Max. process+thread count | 599 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/2/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group =
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840764#comment-17840764
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 commented on code in PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#discussion_r1579314396
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:
##
@@ -242,8 +242,8 @@ static int write_pid_to_cgroup_as_root(const char*
cgroup_file, pid_t pid) {
strerror(errno));
rc = -1;
goto cleanup;
- } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
-fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
cgroup_file);
+ } else if (buf.f_type != CGROUP_SUPER_MAGIC || buf.f_type !=
CGROUP2_SUPER_MAGIC) {
Review Comment:
Thanks, this should be the reason. Added.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840705#comment-17840705
]
ASF GitHub Bot commented on YARN-11690:
---
tomicooler commented on code in PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#discussion_r1579096249
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:
##
@@ -242,8 +242,8 @@ static int write_pid_to_cgroup_as_root(const char*
cgroup_file, pid_t pid) {
strerror(errno));
rc = -1;
goto cleanup;
- } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
-fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
cgroup_file);
+ } else if (buf.f_type != CGROUP_SUPER_MAGIC || buf.f_type !=
CGROUP2_SUPER_MAGIC) {
Review Comment:
https://docs.kernel.org/admin-guide/cgroup-v2.html
It should come from
https://github.com/torvalds/linux/blob/v4.6/include/uapi/linux/magic.h#L58
which we include, probably it is an old kernel.
Could be fixed with:
```
#ifndef CGROUP2_SUPER_MAGIC
#define CGROUP2_SUPER_MAGIC 0x63677270
#endif
```
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840570#comment-17840570
]
ASF GitHub Bot commented on YARN-11690:
---
hadoop-yetus commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2075790931
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 44m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 48s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 91m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| -1 :x: | compile | 0m 47s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | cc | 0m 47s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | golang | 0m 47s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | javac | 0m 47s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 50s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 181m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6771 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux b41f66700d8a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a3beaf9d526f13c4cd22b9e8ee1de470a4462c5a |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/1/testReport/ |
| Max. process+thread count | 532 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840526#comment-17840526
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 opened a new pull request, #6771:
URL: https://github.com/apache/hadoop/pull/6771
### Description of PR
The container executor function write_pid_to_cgroup_as_root writes the PID
of the newly launched container to the correct cgroup.procs file. However it
checks if the file is mounted on a cgroup filesystem, and does that check using
the magic number, which differs for v1 and v2. This should handle v1 or v2
filesystems as well.
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, ) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
