Yicong Huang created SPARK-54929:
------------------------------------
Summary: taskContext._resources reset in loop causes only last
resource to be kept
Key: SPARK-54929
URL: https://issues.apache.org/jira/browse/SPARK-54929
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.1.0
Reporter: Yicong Huang
In {{worker.py}}, {{taskContext._resources}} is reset inside the for loop, so
only the last resource is kept when multiple resources exist.
{code:python}
taskContext._resources = {}
for r in range(read_int(infile)):
key = utf8_deserializer.loads(infile)
name = utf8_deserializer.loads(infile)
addresses = []
taskContext._resources = {} # should be removed
for a in range(read_int(infile)):
addresses.append(utf8_deserializer.loads(infile))
taskContext._resources[key] = ResourceInformation(name, addresses)
{code}
This was introduced in SPARK-28234 (2019). Existing tests only cover
single-resource case so the bug was not caught.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
