David Figueroa created SPARK-24092: -------------------------------------- Summary: spark.python.worker.reuse does not work? Key: SPARK-24092 URL: https://issues.apache.org/jira/browse/SPARK-24092 Project: Spark Issue Type: Question Components: PySpark Affects Versions: 2.3.0 Reporter: David Figueroa
{{spark.python.worker.reuse is true by default but even after explicitly setting to true the code below does not print the same python worker process ids.}} {{{{{code:title=procid.py|borderStyle=solid} }}}} def return_pid(_): yield os.getpid() spark = SparkSession.builder.getOrCreate() pids = set(spark.sparkContext.range(32).mapPartitions(return_pid).collect()) print(pids) pids = set(spark.sparkContext.range(32).mapPartitions(return_pid).collect()) print(pids) {{{{{code}}}}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org