comphead commented on code in PR #2103: URL: https://github.com/apache/datafusion-comet/pull/2103#discussion_r2268319737
########## docs/source/contributor-guide/benchmarking_macos.md: ########## @@ -82,6 +72,17 @@ export SPARK_MASTER=spark://Rustys-MacBook-Pro.local:7077 $SPARK_HOME/sbin/start-worker.sh $SPARK_MASTER ``` +### Start local Apache Spark cluster using `spark-class` +For Apache Spark distributions installed using `brew` tool, it may happen there is no `$SPARK_HOME/sbin` folder on your machine. +In order to start local Apache Spark cluster on `localhost:7077` port, run: +```shell +$SPARK_HOME/bin/spark-class org.apache.spark.deploy.master.Master --host 127.0.0.1 --port 7077 --webui-port 8080 +``` + +Once master has started, in separate console start the worker referring the spark master uri on `localhost:7077` +```shell +$SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker --cores 8 --memory 16G spark://localhost:7077 Review Comment: We have to set it up in advance, otherwise the worker takes all available resources on its start This is from my machine, if I start the worker without specifying resources ``` 25/08/11 17:51:22 INFO Worker: Starting Spark worker 10.1.104.58:62006 with 16 cores, 63.0 GiB RAM ``` And with resources specified ``` 25/08/11 17:52:59 INFO Worker: Starting Spark worker 10.1.104.58:62009 with 8 cores, 16.0 GiB RAM ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
