yoda-mon opened a new pull request, #907:
URL: https://github.com/apache/bigtop/pull/907
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'BIGTOP-3638: Your PR title ...'.
-->
### Description of PR
Add kubernetes related libraries to spark package.
### How was this patch tested?
1. Check the libraries are included
```sh
# On CentOS7
$ ./gradlew spark-pkg
$ rpm -qlp output/spark/noarch/spark-core-3.1.2-1.el7.noarch.rpm| grep kube
/usr/lib/spark/jars/kubernetes-client-4.12.0.jar
/usr/lib/spark/jars/kubernetes-model-admissionregistration-4.12.0.jar
...
/usr/lib/spark/kubernetes/tests/worker_memory_check.py
# On Ubuntu 18.04
$ ./gradlew spark-pkg
dpkg -c output/spark/spark-core_3.1.2-1_all.deb |grep kube
-rw-r--r-- root/root 951167 2022-05-27 08:59
./usr/lib/spark/jars/kubernetes-client-4.12.0.jar
-rw-r--r-- root/root 836222 2022-05-27 08:59
./usr/lib/spark/jars/kubernetes-model-admissionregistration-4.12.0.jar
...
-rw-r--r-- root/root 1585 2022-05-27 08:59
./usr/lib/spark/kubernetes/tests/worker_memory_check.py
```
2. Check whether it works
```
# After Installing built RPMs on CentOS7 and running kubernetes by kind
$ spark-shell \
--master k8s://https://127.0.0.1:37805 \
--deploy-mode client \
--conf spark.executor.instances=1 \
--conf spark.kubernetes.container.image=apache/spark:v3.1.3
scala> val data = Seq(("Java", "20000"), ("Python", "100000"), ("Scala",
"3000")).toDF.show()
[Stage 0:> (0 + 1
+------+------+
| _1| _2|
+------+------+
| Java| 20000|
|Python|100000|
| Scala| 3000|
+------+------+
```
3. Check it does not affect the existed features
```
# On CentOS7
./gradlew \
docker-provisioner \
-POS=centos-7 \
-Pnum_instances=3 \
-Pmemory=8g \
-Penable_local_repo \
-Pstack="hdfs,yarn,mapreduce,spark" \
-Psmoke_tests="spark"
# On Ubuntu 18.04
./gradlew \
docker-provisioner \
-POS=ubuntu-18.04 \
-Pnum_instances=3 \
-Pmemory=8g \
-Penable_local_repo \
-Pstack="hdfs,yarn,mapreduce,spark" \
-Psmoke_tests="spark"
```
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'BIGTOP-3638. Your PR title ...')?
- [ ] Make sure that newly added files do not have any licensing issues.
When in doubt refer to https://www.apache.org/licenses/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
