masatana opened a new pull request, #1273:
URL: https://github.com/apache/bigtop/pull/1273
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'BIGTOP-3638: Your PR title ...'.
-->
### Description of PR
https://issues.apache.org/jira/browse/BIGTOP-4112
We don't have to specify `PYSPARK_PYTHON` in `/usr/bin/pyspark` as it just
execute `exec /usr/lib/spark/bin/pyspark` and
`/usr/lib/spark/bin/pyspark` handles `PYSPARK_PYTHON` like this.
https://github.com/apache/spark/blob/v3.3.4/bin/pyspark#L41-L43
```
if [[ -z "$PYSPARK_PYTHON" ]]; then
PYSPARK_PYTHON=python3
fi
```
### How was this patch tested?
On rocky-8
```
$ ./gradlew spark-clean spark-pkg
$ (install built rpm packages)
$ pyspark
[GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] on linux
Type "help", "copyright", "credits" or "license" for more information.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/spark/jars/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
24/05/27 14:56:57 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.3.4
/_/
Using Python version 3.6.8 (default, Jan 15 2024 23:09:02)
Spark context Web UI available at
http://ip-172-31-4-21.ap-northeast-1.compute.internal:4040
Spark context available as 'sc' (master = local[*], app id =
local-1716821818171).
SparkSession available as 'spark'.
>>>
```
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'BIGTOP-3638. Your PR title ...')?
- [x] Make sure that newly added files do not have any licensing issues.
When in doubt refer to https://www.apache.org/licenses/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
