chengshiwen opened a new issue #5035:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/5035
**Describe the question**
The `HOME` environment variables of multiple executors are as follows:
```
export HADOOP_HOME=/opt/soft/hadoop
export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop
export SPARK_HOME1=/opt/soft/spark1
export SPARK_HOME2=/opt/soft/spark2
export PYTHON_HOME=/opt/soft/python
export JAVA_HOME=/opt/soft/java
export HIVE_HOME=/opt/soft/hive
export FLINK_HOME=/opt/soft/flink
export DATAX_HOME=/opt/soft/datax/bin/datax.py
export
PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH:$FLINK_HOME/bin:$DATAX_HOME:$PATH
```
But the value of some `HOME` variables is not reasonable, like `PYTHON_HOME`
and `DATAX_HOME`. Both of them are the file path, so `$PYTHON_HOME` and
`$DATAX_HOME` in `PATH` will not take effect. Related issues: #5018 and #5024.
At the same time, for multiple versions of an executor, we will encounter
scalability problems, like `SPARK_HOME1` and `SPARK_HOME2`, since the Spark
3.0.0 has released on June 18, 2020 (Now the latest version is 3.1.1). And
`$SPARK_HOME1/bin:$SPARK_HOME2/bin` in `PATH` can lead to influence each other
So, we need to think about the management and specification of `HOME`
environment variables for multiple executors (maybe with multiple versions).
**What are the current deficiencies and the benefits of improvement**
Specifically, as shown below:
- The `HOME` environment variables of all executors should use the executor
root directory instead of the binary path
- There are two solutions for multiple versions of an executor:
- Just use **ONE** `HOME` environment variable for an executor, so
`SPARK_HOME1` and `SPARK_HOME2` can be integrated into `SPARK_HOME`. Use
`worker.groups` to distinguish different versions of an executor. for example,
set `SPARK_HOME=/opt/spark-1.6.3` on node1 and `SPARK_HOME=/opt/spark-2.4.7` on
node2, we only set `worker.groups=spark1` on node1 and `worker.groups=spark2`
on node2. The cost of this solution is small.
- Use a table in the database to manage multiple versions like
`t_ds_executor_definition`, and it's managed and edited by user. The table
schema contains `task type`, `executor name`, `executor HOME path` and etc.
This solution has very good scalability.
**Which version of DolphinScheduler:**
-[dev]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
