Re: Spark Connect, Master, and Workers

Hi Kezhi, Yes, you no longer need to start a master to make the client work. Please see the quickstart. https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_connect.html You can think of Spark Connect as an API on top of Master so workers can be added to the cluster same

Re: Contributing to Spark MLLib

Good morning Dipayan, Happy to see another contributor! Please go through this document for contributors. Please note the MLlib-specific contribution guidelines section in particular. https://spark.apache.org/contributing.html Since you are looking for something to start with, take a look at

Re: PySpark error java.lang.IllegalArgumentException

Hi Khalid,Elango mentioned the file is working fine in our another environment with the same driver and executor memoryBrianOn Jul 7, 2023, at 10:18 AM, Khalid Mammadov wrote:Perhaps that parquet file is corrupted or got that is in that folder?To check, try to read that file with pandas or other