Re: PySpark cores

Hi, Agree with above response, but in case you are using arrow and transferring data from JVM to python and back, then please try to check how are things getting executed in python. Please let me know what is the processing you are trying to do while using arrow. Regards, Gourav Sengupta On

[no subject]

unsubscribe [Impelsys] Impelsys Disclaimer The information contained in this message is intended for the addresseeonly and may contain classified information. If you are not the addressee,please delete this message and notify the sender; you should not copy or

Re: PySpark cores

I think you are looking for the spark.task.cpus configuration parameter. Op vr 29 jul. 2022 om 07:41 schreef Andrew Melo : > Hello, > > Is there a way to tell Spark that PySpark (arrow) functions use > multiple cores? If we have an executor with 8 cores, we would like to > have a single PySpark