masatana opened a new pull request, #1317:
URL: https://github.com/apache/bigtop/pull/1317
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'BIGTOP-3638: Your PR title ...'.
-->
### Description of PR
https://issues.apache.org/jira/browse/BIGTOP-4279
This is a draft PR because we need to merge #1313
### How was this patch tested?
Tested on both RPM(rocky linux8) and DEB (Ubuntu 22.04 w/
https://github.com/apache/bigtop/pull/1304,
https://github.com/apache/bigtop/pull/1305)
* Check if building package is SUCCESSFUL
* Check if `yarn` smoke test is SUCCESSFUL
* Check if hadoop-yarn-{resourcemanager, nodemanager, proxyserver, router,
timelineserver} is running with the packages
* Check if the services can start after restating
* Since `/run/hadoop-yarn` is shared by multiple services, I used
`systemd-tmpfile` to create it (Unlike hadoop-mapreduce's approach)
<details>
<summary>Tested logs</summary>
* build & smoketests
```
$ ./gradlew allclean hadoop-pkg-ind repo-ind -POS=rockylinux-8
$ cd provisioner/docker
$ ./docker-hadoop.sh --enable-local-repo --disable-gpg-check
--docker-compose-plugin -C config_rockylinux-8.yaml -F
docker-compose-cgroupv2.yml --stack hdfs,yarn --smoke-tests yarn -c 1
(snip)
BUILD SUCCESSFUL in 15s
28 actionable tasks: 7 executed, 21 up-to-date
Stopped 1 worker daemon(s).
+ rm -rf buildSrc/build/test-results/binary
+ rm -rf /bigtop-home/.gradle
```
* install additional packages (timelineserver, router), `systemctl start`,
then `systemctl status`
```
$ ./docker-hadoop.sh -dcp --exec 1 /bin/bash
[root@a60f5b5943d3 /]# dnf install hadoop-yarn-timelineserver
hadoop-yarn-router
(snip)
Complete!
[root@a60f5b5943d3 /]# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl start
hadoop-yarn-$service_name; done
[root@a60f5b5943d3 /]# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl status --no-pager
hadoop-yarn-$service_name; done
● hadoop-yarn-resourcemanager.service - Hadoop ResourceManager
Loaded: loaded
(/usr/lib/systemd/system/hadoop-yarn-resourcemanager.service; static; vendor
preset: disabled)
Active: active (running) since Sun 2024-12-01 03:01:49 UTC; 4min 55s ago
Docs: https://hadoop.apache.org/
Main PID: 6692 (java)
Tasks: 227 (limit: 5099)
Memory: 437.7M
CGroup: /system.slice/hadoop-yarn-resourcemanager.service
└─6692
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.432.b06-2.el8.aarch64/bin/java
-Dproc_resourcemanager -Djava.net.preferIPv4Stack=true
-Dservice.libdir=/usr/lib/hadoop-yarn/./,/usr/lib/hadoop-yarn/lib,/usr/lib/hadoop-hdfs/./,/usr/lib/hadoop-hdfs/lib,/usr/lib/hadoop/./,/usr/lib/hadoop/lib
-Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=hadoop-yarn-res…
Dec 01 03:01:47 a60f5b5943d3 systemd[1]: Starting Hadoop ResourceManager...
Dec 01 03:01:49 a60f5b5943d3 systemd[1]: Started Hadoop ResourceManager.
● hadoop-yarn-nodemanager.service - Hadoop NodeManager
Loaded: loaded (/usr/lib/systemd/system/hadoop-yarn-nodemanager.service;
static; vendor preset: disabled)
Active: active (running) since Sun 2024-12-01 03:02:21 UTC; 4min 24s ago
Docs: https://hadoop.apache.org/
Main PID: 8309 (java)
Tasks: 102 (limit: 5099)
Memory: 293.2M
CGroup: /system.slice/hadoop-yarn-nodemanager.service
└─8309
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.432.b06-2.el8.aarch64/bin/java
-Dproc_nodemanager -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn
-Dyarn.log.file=hadoop-yarn-nodemanager-a60f5b5943d3.log
-Dyarn.home.dir=/usr/lib/hadoop-yarn -Dyarn.root.logger=INFO,console
-Djava.library.path=//usr/lib/hadoop/lib/native -Dhadoop.log.di…
Dec 01 03:02:19 a60f5b5943d3 systemd[1]: Starting Hadoop NodeManager...
Dec 01 03:02:21 a60f5b5943d3 systemd[1]: Started Hadoop NodeManager.
● hadoop-yarn-proxyserver.service - Hadoop ProxyServer
Loaded: loaded (/usr/lib/systemd/system/hadoop-yarn-proxyserver.service;
static; vendor preset: disabled)
Active: active (running) since Sun 2024-12-01 03:01:56 UTC; 4min 48s ago
Docs: https://hadoop.apache.org/
Main PID: 7184 (java)
Tasks: 33 (limit: 5099)
Memory: 129.3M
CGroup: /system.slice/hadoop-yarn-proxyserver.service
└─7184
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.432.b06-2.el8.aarch64/bin/java
-Dproc_proxyserver -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn
-Dyarn.log.file=hadoop-yarn-proxyserver-a60f5b5943d3.log
-Dyarn.home.dir=/usr/lib/hadoop-yarn -Dyarn.root.logger=INFO,console
-Djava.library.path=//usr/lib/hadoop/lib/native -Dhadoop.log.di…
Dec 01 03:01:54 a60f5b5943d3 systemd[1]: Starting Hadoop ProxyServer...
Dec 01 03:01:56 a60f5b5943d3 systemd[1]: Started Hadoop ProxyServer.
● hadoop-yarn-timelineserver.service - Hadoop NodeManager
Loaded: loaded
(/usr/lib/systemd/system/hadoop-yarn-timelineserver.service; static; vendor
preset: disabled)
Active: active (running) since Sun 2024-12-01 03:06:27 UTC; 18s ago
Docs: https://hadoop.apache.org/
Process: 9484 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start timelineserver (code=exited, status=0/SUCCESS)
Main PID: 9535 (java)
Tasks: 53 (limit: 5099)
Memory: 424.5M
CGroup: /system.slice/hadoop-yarn-timelineserver.service
└─9535
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.432.b06-2.el8.aarch64/bin/java
-Dproc_timelineserver -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn
-Dyarn.log.file=hadoop-yarn-timelineserver-a60f5b5943d3.log
-Dyarn.home.dir=/usr/lib/hadoop-yarn -Dyarn.root.logger=INFO,console
-Djava.library.path=//usr/lib/hadoop/lib/native -Dhadoop.…
Dec 01 03:06:25 a60f5b5943d3 systemd[1]: Starting Hadoop NodeManager...
Dec 01 03:06:27 a60f5b5943d3 systemd[1]: Started Hadoop NodeManager.
● hadoop-yarn-router.service - Hadoop Router
Loaded: loaded (/usr/lib/systemd/system/hadoop-yarn-router.service;
static; vendor preset: disabled)
Active: active (running) since Sun 2024-12-01 03:06:29 UTC; 16s ago
Docs: https://hadoop.apache.org/
Process: 9598 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start router (code=exited, status=0/SUCCESS)
Main PID: 9649 (java)
Tasks: 97 (limit: 5099)
Memory: 274.4M
CGroup: /system.slice/hadoop-yarn-router.service
└─9649
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.432.b06-2.el8.aarch64/bin/java
-Dproc_router -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn
-Dyarn.log.file=hadoop-yarn-router-a60f5b5943d3.log
-Dyarn.home.dir=/usr/lib/hadoop-yarn -Dyarn.root.logger=INFO,console
-Djava.library.path=//usr/lib/hadoop/lib/native -Dhadoop.log.dir=/var/log…
Dec 01 03:06:27 a60f5b5943d3 systemd[1]: Starting Hadoop Router...
Dec 01 03:06:29 a60f5b5943d3 systemd[1]: Started Hadoop Router.
```
* Check if prepared unit files are used w/ `systemctl cat`
```
[root@a60f5b5943d3 /]# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl cat hadoop-yarn-$service_name;
done
# /usr/lib/systemd/system/hadoop-yarn-resourcemanager.service
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Documentation=https://hadoop.apache.org/
Description=Hadoop ResourceManager
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
User=yarn
Group=yarn
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/run/hadoop-yarn/hadoop-yarn-resourcemanager.pid
SuccessExitStatus=5 6
ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon start
resourcemanager
ExecStop=/usr/bin/yarn --config /etc/hadoop/conf --daemon stop
resourcemanager
# /usr/lib/systemd/system/hadoop-yarn-nodemanager.service
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Documentation=https://hadoop.apache.org/
Description=Hadoop NodeManager
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
User=yarn
Group=yarn
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/run/hadoop-yarn/hadoop-yarn-nodemanager.pid
SuccessExitStatus=5 6
ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon start nodemanager
ExecStop=/usr/bin/yarn --config /etc/hadoop/conf --daemon stop nodemanager
# /usr/lib/systemd/system/hadoop-yarn-proxyserver.service
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Documentation=https://hadoop.apache.org/
Description=Hadoop ProxyServer
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
User=yarn
Group=yarn
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/run/hadoop-yarn/hadoop-yarn-proxyserver.pid
SuccessExitStatus=5 6
ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon start proxyserver
ExecStop=/usr/bin/yarn --config /etc/hadoop/conf --daemon stop proxyserver
# /usr/lib/systemd/system/hadoop-yarn-timelineserver.service
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Documentation=https://hadoop.apache.org/
Description=Hadoop NodeManager
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
User=yarn
Group=yarn
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/run/hadoop-yarn/hadoop-yarn-timelineserver.pid
SuccessExitStatus=5 6
ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon start
timelineserver
ExecStop=/usr/bin/yarn --config /etc/hadoop/conf --daemon stop timelineserver
# /usr/lib/systemd/system/hadoop-yarn-router.service
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Documentation=https://hadoop.apache.org/
Description=Hadoop Router
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
User=yarn
Group=yarn
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/run/hadoop-yarn/hadoop-yarn-router.pid
SuccessExitStatus=5 6
ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon start router
ExecStop=/usr/bin/yarn --config /etc/hadoop/conf --daemon stop router
```
* restart container
```
[root@a60f5b5943d3 /]# exit
$ docker restart a60f5b5943d3
$ ./docker-hadoop.sh -dcp --exec 1 /bin/bash
[root@a60f5b5943d3 /]# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl start
hadoop-yarn-$service_name; done
[root@a60f5b5943d3 /]# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl status --no-pager
hadoop-yarn-$service_name; done
● hadoop-yarn-resourcemanager.service - Hadoop ResourceManager
Loaded: loaded
(/usr/lib/systemd/system/hadoop-yarn-resourcemanager.service; static; vendor
preset: disabled)
Active: active (running) since Sun 2024-12-01 03:12:27 UTC; 14s ago
Docs: https://hadoop.apache.org/
Process: 401 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start resourcemanager (code=exited, status=0/SUCCESS)
Main PID: 455 (java)
Tasks: 231 (limit: 5099)
Memory: 456.5M
CGroup: /system.slice/hadoop-yarn-resourcemanager.service
└─455
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.432.b06-2.el8.aarch64/bin/java
-Dproc_resourcemanager -Djava.net.preferIPv4Stack=true
-Dservice.libdir=/usr/lib/hadoop-yarn/./,/usr/lib/hadoop-yarn/lib,/usr/lib/hadoop-hdfs/./,/usr/lib/hadoop-hdfs/lib,/usr/lib/hadoop/./,/usr/lib/hadoop/lib
-Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=hadoop-yarn-reso…
Dec 01 03:12:25 a60f5b5943d3 systemd[1]: Starting Hadoop ResourceManager...
Dec 01 03:12:27 a60f5b5943d3 systemd[1]: Started Hadoop ResourceManager.
● hadoop-yarn-nodemanager.service - Hadoop NodeManager
Loaded: loaded (/usr/lib/systemd/system/hadoop-yarn-nodemanager.service;
static; vendor preset: disabled)
Active: active (running) since Sun 2024-12-01 03:12:29 UTC; 12s ago
Docs: https://hadoop.apache.org/
Process: 695 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start nodemanager (code=exited, status=0/SUCCESS)
Main PID: 749 (java)
Tasks: 107 (limit: 5099)
Memory: 287.7M
CGroup: /system.slice/hadoop-yarn-nodemanager.service
└─749
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.432.b06-2.el8.aarch64/bin/java
-Dproc_nodemanager -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn
-Dyarn.log.file=hadoop-yarn-nodemanager-a60f5b5943d3.log
-Dyarn.home.dir=/usr/lib/hadoop-yarn -Dyarn.root.logger=INFO,console
-Djava.library.path=//usr/lib/hadoop/lib/native -Dhadoop.log.dir…
Dec 01 03:12:27 a60f5b5943d3 systemd[1]: Starting Hadoop NodeManager...
Dec 01 03:12:29 a60f5b5943d3 systemd[1]: Started Hadoop NodeManager.
● hadoop-yarn-proxyserver.service - Hadoop ProxyServer
Loaded: loaded (/usr/lib/systemd/system/hadoop-yarn-proxyserver.service;
static; vendor preset: disabled)
Active: active (running) since Sun 2024-12-01 03:12:31 UTC; 10s ago
Docs: https://hadoop.apache.org/
Process: 918 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start proxyserver (code=exited, status=0/SUCCESS)
Main PID: 970 (java)
Tasks: 34 (limit: 5099)
Memory: 123.6M
CGroup: /system.slice/hadoop-yarn-proxyserver.service
└─970
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.432.b06-2.el8.aarch64/bin/java
-Dproc_proxyserver -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn
-Dyarn.log.file=hadoop-yarn-proxyserver-a60f5b5943d3.log
-Dyarn.home.dir=/usr/lib/hadoop-yarn -Dyarn.root.logger=INFO,console
-Djava.library.path=//usr/lib/hadoop/lib/native -Dhadoop.log.dir…
Dec 01 03:12:29 a60f5b5943d3 systemd[1]: Starting Hadoop ProxyServer...
Dec 01 03:12:31 a60f5b5943d3 systemd[1]: Started Hadoop ProxyServer.
● hadoop-yarn-timelineserver.service - Hadoop NodeManager
Loaded: loaded
(/usr/lib/systemd/system/hadoop-yarn-timelineserver.service; static; vendor
preset: disabled)
Active: active (running) since Sun 2024-12-01 03:12:33 UTC; 8s ago
Docs: https://hadoop.apache.org/
Process: 1012 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start timelineserver (code=exited, status=0/SUCCESS)
Main PID: 1064 (java)
Tasks: 53 (limit: 5099)
Memory: 246.1M
CGroup: /system.slice/hadoop-yarn-timelineserver.service
└─1064
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.432.b06-2.el8.aarch64/bin/java
-Dproc_timelineserver -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn
-Dyarn.log.file=hadoop-yarn-timelineserver-a60f5b5943d3.log
-Dyarn.home.dir=/usr/lib/hadoop-yarn -Dyarn.root.logger=INFO,console
-Djava.library.path=//usr/lib/hadoop/lib/native -Dhadoop.…
Dec 01 03:12:31 a60f5b5943d3 systemd[1]: Starting Hadoop NodeManager...
Dec 01 03:12:33 a60f5b5943d3 systemd[1]: Started Hadoop NodeManager.
● hadoop-yarn-router.service - Hadoop Router
Loaded: loaded (/usr/lib/systemd/system/hadoop-yarn-router.service;
static; vendor preset: disabled)
Active: active (running) since Sun 2024-12-01 03:12:36 UTC; 6s ago
Docs: https://hadoop.apache.org/
Process: 1127 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start router (code=exited, status=0/SUCCESS)
Main PID: 1179 (java)
Tasks: 97 (limit: 5099)
Memory: 277.0M
CGroup: /system.slice/hadoop-yarn-router.service
└─1179
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.432.b06-2.el8.aarch64/bin/java
-Dproc_router -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn
-Dyarn.log.file=hadoop-yarn-router-a60f5b5943d3.log
-Dyarn.home.dir=/usr/lib/hadoop-yarn -Dyarn.root.logger=INFO,console
-Djava.library.path=//usr/lib/hadoop/lib/native -Dhadoop.log.dir=/var/log…
Dec 01 03:12:33 a60f5b5943d3 systemd[1]: Starting Hadoop Router...
Dec 01 03:12:36 a60f5b5943d3 systemd[1]: Started Hadoop Router.
```
#### DEB (Ubuntu-22.04)
* build & smoketests
```
$ ./gradlew hadoop-clean bigtop-utils-pkg bigtop-jsvc-pkg bigtop-groovy-pkg
hadoop-pkg repo
(snip)
$ cd provisioner/docker/
$ ./docker-hadoop.sh --enable-local-repo --disable-gpg-check
--docker-compose-plugin -C config_ubuntu-22.04.yaml -F
docker-compose-cgroupv2.yml --stack hdfs,yarn --smoke-tests yarn -c 1
(snip)
Gradle Test Executor 2 finished executing tests.
> Task :bigtop-tests:smoke-tests:yarn:test
Finished generating test XML results (0.003 secs) into:
/bigtop-home/bigtop-tests/smoke-tests/yarn/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.006 secs) into:
/bigtop-home/bigtop-tests/smoke-tests/yarn/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:yarn:test (Thread[Execution worker for ':' Thread
6,5,main]) completed. Took 2.93 secs.
BUILD SUCCESSFUL in 20s
28 actionable tasks: 6 executed, 22 up-to-date
Stopped 1 worker daemon(s).
+ rm -rf buildSrc/build/test-results/binary
+ rm -rf /bigtop-home/.gradle
```
* install additional packages (timelineserver, router), `systemctl start`,
then `systemctl status`
```
$ ./docker-hadoop.sh -dcp --exec 1 /bin/bash
root@cd7a61f76c23:/# apt install hadoop-yarn-timelineserver
hadoop-yarn-router
(snip)
root@cd7a61f76c23:/# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl start
hadoop-yarn-$service_name; done
root@cd7a61f76c23:/# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl status --no-pager
hadoop-yarn-$service_name; done
● hadoop-yarn-resourcemanager.service - Hadoop ResourceManager
Loaded: loaded
(/lib/systemd/system/hadoop-yarn-resourcemanager.service; static)
Active: active (running) since Sun 2024-12-01 09:20:23 UTC; 5min ago
Docs: https://hadoop.apache.org/
Main PID: 5842 (java)
Tasks: 227 (limit: 956)
Memory: 422.4M
CPU: 7.317s
CGroup: /system.slice/hadoop-yarn-resourcemanager.service
└─5842 /usr/lib/jvm/java-1.8.0-openjdk-arm64/bin/java
-Dproc_resourcemanager -Djava.net.preferIPv4Stack=true
-Dservice.libdir=/usr/lib/hadoop-yarn/./,/usr/lib/hado…
Dec 01 09:20:21 cd7a61f76c23 systemd[1]: Starting Hadoop ResourceManager...
Dec 01 09:20:23 cd7a61f76c23 systemd[1]: Started Hadoop ResourceManager.
● hadoop-yarn-nodemanager.service - Hadoop NodeManager
Loaded: loaded (/lib/systemd/system/hadoop-yarn-nodemanager.service;
static)
Active: active (running) since Sun 2024-12-01 09:20:44 UTC; 5min ago
Docs: https://hadoop.apache.org/
Main PID: 6949 (java)
Tasks: 102 (limit: 956)
Memory: 282.3M
CPU: 5.484s
CGroup: /system.slice/hadoop-yarn-nodemanager.service
└─6949 /usr/lib/jvm/java-1.8.0-openjdk-arm64/bin/java
-Dproc_nodemanager -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=hadoop…
Dec 01 09:20:42 cd7a61f76c23 systemd[1]: Starting Hadoop NodeManager...
Dec 01 09:20:44 cd7a61f76c23 systemd[1]: Started Hadoop NodeManager.
● hadoop-yarn-proxyserver.service - Hadoop ProxyServer
Loaded: loaded (/lib/systemd/system/hadoop-yarn-proxyserver.service;
static)
Active: active (running) since Sun 2024-12-01 09:20:29 UTC; 5min ago
Docs: https://hadoop.apache.org/
Main PID: 6429 (java)
Tasks: 33 (limit: 956)
Memory: 118.4M
CPU: 1.318s
CGroup: /system.slice/hadoop-yarn-proxyserver.service
└─6429 /usr/lib/jvm/java-1.8.0-openjdk-arm64/bin/java
-Dproc_proxyserver -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=hadoop…
Dec 01 09:20:27 cd7a61f76c23 systemd[1]: Starting Hadoop ProxyServer...
Dec 01 09:20:29 cd7a61f76c23 systemd[1]: Started Hadoop ProxyServer.
● hadoop-yarn-timelineserver.service - Hadoop NodeManager
Loaded: loaded (/lib/systemd/system/hadoop-yarn-timelineserver.service;
static)
Active: active (running) since Sun 2024-12-01 09:26:15 UTC; 7s ago
Docs: https://hadoop.apache.org/
Process: 8047 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start timelineserver (code=exited, status=0/SUCCESS)
Main PID: 8086 (java)
Tasks: 53 (limit: 956)
Memory: 351.3M
CPU: 2.539s
CGroup: /system.slice/hadoop-yarn-timelineserver.service
└─8086 /usr/lib/jvm/java-1.8.0-openjdk-arm64/bin/java
-Dproc_timelineserver -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=had…
Dec 01 09:26:13 cd7a61f76c23 systemd[1]: Starting Hadoop NodeManager...
Dec 01 09:26:15 cd7a61f76c23 systemd[1]: Started Hadoop NodeManager.
● hadoop-yarn-router.service - Hadoop Router
Loaded: loaded (/lib/systemd/system/hadoop-yarn-router.service; static)
Active: active (running) since Sun 2024-12-01 09:26:17 UTC; 5s ago
Docs: https://hadoop.apache.org/
Process: 8149 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start router (code=exited, status=0/SUCCESS)
Main PID: 8188 (java)
Tasks: 97 (limit: 956)
Memory: 259.5M
CPU: 2.636s
CGroup: /system.slice/hadoop-yarn-router.service
└─8188 /usr/lib/jvm/java-1.8.0-openjdk-arm64/bin/java
-Dproc_router -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=hadoop-yarn…
Dec 01 09:26:15 cd7a61f76c23 systemd[1]: Starting Hadoop Router...
Dec 01 09:26:17 cd7a61f76c23 systemd[1]: Started Hadoop Router.
```
* * Check if prepared unit files are used w/ `systemctl cat` `systemctl cat`
```
root@cd7a61f76c23:/# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl cat hadoop-yarn-$service_name;
done
# /lib/systemd/system/hadoop-yarn-resourcemanager.service
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Documentation=https://hadoop.apache.org/
Description=Hadoop ResourceManager
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
User=yarn
Group=yarn
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/run/hadoop-yarn/hadoop-yarn-resourcemanager.pid
SuccessExitStatus=5 6
ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon start
resourcemanager
ExecStop=/usr/bin/yarn --config /etc/hadoop/conf --daemon stop
resourcemanager
# /lib/systemd/system/hadoop-yarn-nodemanager.service
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Documentation=https://hadoop.apache.org/
Description=Hadoop NodeManager
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
User=yarn
Group=yarn
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/run/hadoop-yarn/hadoop-yarn-nodemanager.pid
SuccessExitStatus=5 6
ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon start nodemanager
ExecStop=/usr/bin/yarn --config /etc/hadoop/conf --daemon stop nodemanager
# /lib/systemd/system/hadoop-yarn-proxyserver.service
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Documentation=https://hadoop.apache.org/
Description=Hadoop ProxyServer
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
User=yarn
Group=yarn
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/run/hadoop-yarn/hadoop-yarn-proxyserver.pid
SuccessExitStatus=5 6
ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon start proxyserver
ExecStop=/usr/bin/yarn --config /etc/hadoop/conf --daemon stop proxyserver
# /lib/systemd/system/hadoop-yarn-timelineserver.service
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Documentation=https://hadoop.apache.org/
Description=Hadoop NodeManager
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
User=yarn
Group=yarn
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/run/hadoop-yarn/hadoop-yarn-timelineserver.pid
SuccessExitStatus=5 6
ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon start
timelineserver
ExecStop=/usr/bin/yarn --config /etc/hadoop/conf --daemon stop timelineserver
# /lib/systemd/system/hadoop-yarn-router.service
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Documentation=https://hadoop.apache.org/
Description=Hadoop Router
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
User=yarn
Group=yarn
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/run/hadoop-yarn/hadoop-yarn-router.pid
SuccessExitStatus=5 6
ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon start router
ExecStop=/usr/bin/yarn --config /etc/hadoop/conf --daemon stop router
```
* restart container
```
root@cd7a61f76c23:/# exit
exit
masatana@ubuntu-2204-dev:~/bigtop/provisioner/docker$ docker restart
cd7a61f76c23
cd7a61f76c23
masatana@ubuntu-2204-dev:~/bigtop/provisioner/docker$ ./docker-hadoop.sh
-dcp --exec 1 /bin/bash
root@cd7a61f76c23:/# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl status --no-pager
hadoop-yarn-$service_name; done
○ hadoop-yarn-resourcemanager.service - Hadoop ResourceManager
Loaded: loaded
(/lib/systemd/system/hadoop-yarn-resourcemanager.service; static)
Active: inactive (dead)
Docs: https://hadoop.apache.org/
Dec 01 09:20:21 cd7a61f76c23 systemd[1]: Starting Hadoop ResourceManager...
Dec 01 09:20:23 cd7a61f76c23 systemd[1]: Started Hadoop ResourceManager.
○ hadoop-yarn-nodemanager.service - Hadoop NodeManager
Loaded: loaded (/lib/systemd/system/hadoop-yarn-nodemanager.service;
static)
Active: inactive (dead)
Docs: https://hadoop.apache.org/
Dec 01 09:20:42 cd7a61f76c23 systemd[1]: Starting Hadoop NodeManager...
Dec 01 09:20:44 cd7a61f76c23 systemd[1]: Started Hadoop NodeManager.
○ hadoop-yarn-proxyserver.service - Hadoop ProxyServer
Loaded: loaded (/lib/systemd/system/hadoop-yarn-proxyserver.service;
static)
Active: inactive (dead)
Docs: https://hadoop.apache.org/
Dec 01 09:20:27 cd7a61f76c23 systemd[1]: Starting Hadoop ProxyServer...
Dec 01 09:20:29 cd7a61f76c23 systemd[1]: Started Hadoop ProxyServer.
○ hadoop-yarn-timelineserver.service - Hadoop NodeManager
Loaded: loaded (/lib/systemd/system/hadoop-yarn-timelineserver.service;
static)
Active: inactive (dead)
Docs: https://hadoop.apache.org/
Dec 01 09:26:13 cd7a61f76c23 systemd[1]: Starting Hadoop NodeManager...
Dec 01 09:26:15 cd7a61f76c23 systemd[1]: Started Hadoop NodeManager.
○ hadoop-yarn-router.service - Hadoop Router
Loaded: loaded (/lib/systemd/system/hadoop-yarn-router.service; static)
Active: inactive (dead)
Docs: https://hadoop.apache.org/
Dec 01 09:26:15 cd7a61f76c23 systemd[1]: Starting Hadoop Router...
Dec 01 09:26:17 cd7a61f76c23 systemd[1]: Started Hadoop Router.
root@cd7a61f76c23:/# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl start
hadoop-yarn-$service_name; done
root@cd7a61f76c23:/# for service_name in resourcemanager nodemanager
proxyserver timelineserver router; do systemctl status --no-pager
hadoop-yarn-$service_name; done
● hadoop-yarn-resourcemanager.service - Hadoop ResourceManager
Loaded: loaded
(/lib/systemd/system/hadoop-yarn-resourcemanager.service; static)
Active: active (running) since Sun 2024-12-01 09:29:11 UTC; 13s ago
Docs: https://hadoop.apache.org/
Process: 412 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start resourcemanager (code=exited, status=0/SUCCESS)
Main PID: 454 (java)
Tasks: 229 (limit: 956)
Memory: 402.6M
CPU: 3.940s
CGroup: /system.slice/hadoop-yarn-resourcemanager.service
└─454 /usr/lib/jvm/java-1.8.0-openjdk-arm64/bin/java
-Dproc_resourcemanager -Djava.net.preferIPv4Stack=true
-Dservice.libdir=/usr/lib/hadoop-yarn/./,/usr/lib/hadoo…
Dec 01 09:29:09 cd7a61f76c23 systemd[1]: Starting Hadoop ResourceManager...
Dec 01 09:29:11 cd7a61f76c23 systemd[1]: Started Hadoop ResourceManager.
● hadoop-yarn-nodemanager.service - Hadoop NodeManager
Loaded: loaded (/lib/systemd/system/hadoop-yarn-nodemanager.service;
static)
Active: active (running) since Sun 2024-12-01 09:29:13 UTC; 11s ago
Docs: https://hadoop.apache.org/
Process: 691 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start nodemanager (code=exited, status=0/SUCCESS)
Main PID: 733 (java)
Tasks: 107 (limit: 956)
Memory: 287.1M
CPU: 2.745s
CGroup: /system.slice/hadoop-yarn-nodemanager.service
└─733 /usr/lib/jvm/java-1.8.0-openjdk-arm64/bin/java
-Dproc_nodemanager -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=hadoop-…
Dec 01 09:29:11 cd7a61f76c23 systemd[1]: Starting Hadoop NodeManager...
Dec 01 09:29:13 cd7a61f76c23 systemd[1]: Started Hadoop NodeManager.
● hadoop-yarn-proxyserver.service - Hadoop ProxyServer
Loaded: loaded (/lib/systemd/system/hadoop-yarn-proxyserver.service;
static)
Active: active (running) since Sun 2024-12-01 09:29:15 UTC; 9s ago
Docs: https://hadoop.apache.org/
Process: 902 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start proxyserver (code=exited, status=0/SUCCESS)
Main PID: 942 (java)
Tasks: 34 (limit: 956)
Memory: 123.5M
CPU: 1.060s
CGroup: /system.slice/hadoop-yarn-proxyserver.service
└─942 /usr/lib/jvm/java-1.8.0-openjdk-arm64/bin/java
-Dproc_proxyserver -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=hadoop-…
Dec 01 09:29:13 cd7a61f76c23 systemd[1]: Starting Hadoop ProxyServer...
Dec 01 09:29:15 cd7a61f76c23 systemd[1]: Started Hadoop ProxyServer.
● hadoop-yarn-timelineserver.service - Hadoop NodeManager
Loaded: loaded (/lib/systemd/system/hadoop-yarn-timelineserver.service;
static)
Active: active (running) since Sun 2024-12-01 09:29:17 UTC; 7s ago
Docs: https://hadoop.apache.org/
Process: 984 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start timelineserver (code=exited, status=0/SUCCESS)
Main PID: 1024 (java)
Tasks: 53 (limit: 956)
Memory: 256.2M
CPU: 2.300s
CGroup: /system.slice/hadoop-yarn-timelineserver.service
└─1024 /usr/lib/jvm/java-1.8.0-openjdk-arm64/bin/java
-Dproc_timelineserver -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=had…
Dec 01 09:29:15 cd7a61f76c23 systemd[1]: Starting Hadoop NodeManager...
Dec 01 09:29:17 cd7a61f76c23 systemd[1]: Started Hadoop NodeManager.
● hadoop-yarn-router.service - Hadoop Router
Loaded: loaded (/lib/systemd/system/hadoop-yarn-router.service; static)
Active: active (running) since Sun 2024-12-01 09:29:19 UTC; 5s ago
Docs: https://hadoop.apache.org/
Process: 1087 ExecStart=/usr/bin/yarn --config /etc/hadoop/conf --daemon
start router (code=exited, status=0/SUCCESS)
Main PID: 1127 (java)
Tasks: 97 (limit: 956)
Memory: 299.0M
CPU: 2.505s
CGroup: /system.slice/hadoop-yarn-router.service
└─1127 /usr/lib/jvm/java-1.8.0-openjdk-arm64/bin/java
-Dproc_router -Djava.net.preferIPv4Stack=true
-Dyarn.log.dir=/var/log/hadoop-yarn -Dyarn.log.file=hadoop-yarn…
Dec 01 09:29:17 cd7a61f76c23 systemd[1]: Starting Hadoop Router...
Dec 01 09:29:19 cd7a61f76c23 systemd[1]: Started Hadoop Router.
```
</details>
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'BIGTOP-3638. Your PR title ...')?
- [ ] Make sure that newly added files do not have any licensing issues.
When in doubt refer to https://www.apache.org/licenses/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
