wbo4958 commented on PR #45232:
URL: https://github.com/apache/spark/pull/45232#issuecomment-1963263080
## With dynamic allocation enabled.
``` bash
start-connect-server.sh --master spark://192.168.0.106:7077 \
--jars jars/spark-connect_2.13-4.0.0-SNAPSHOT.jar \
--conf spark.executor.cores=4 \
--conf spark.task.cpus=1 \
--conf spark.dynamicAllocation.enabled=true \
--conf spark.dynamicAllocation.maxExecutors=1 \
```
The above command enables the dynamic allocation and the max executors
required is set to 1 in order to test. And then launch the spark connect
pyspark client by
``` bash
pyspark --remote "sc://localhost"
```
### TaskResourceProfile without any specific executor request information
Test code,
```python
from pyspark.resource import ExecutorResourceRequests, TaskResourceRequests,
ResourceProfileBuilder
def filter_func(iterator):
for pdf in iterator:
yield pdf
df = spark.range(0, 100, 1, 4)
treqs = TaskResourceRequests().cpus(3)
rp = ResourceProfileBuilder().require(treqs).build
df.repartition(3).mapInArrow(lambda iter: iter, df.schema, False,
rp).collect()
```
The rp refers to the TaskResourceProfile without any specific executor
request information, thus the executor information will utilize the default
values from Default ResourceProfile (executor.cores=4).
The above code will require an extra executor which will have the same
`executor.cores/memory` as the default ResourceProfile.
### Different executor request information
```python
from pyspark.resource import ExecutorResourceRequests, TaskResourceRequests,
ResourceProfileBuilder
def filter_func(iterator):
for pdf in iterator:
yield pdf
df = spark.range(0, 100, 1, 4)
ereqs = ExecutorResourceRequests().cores(6)
treqs = TaskResourceRequests().cpus(5)
rp = ResourceProfileBuilder().require(treqs).require(ereqs).build
df.repartition(3).mapInArrow(lambda iter: iter, df.schema, False,
rp).collect()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
