Davies Liu created SPARK-8202: --------------------------------- Summary: PySpark: infinite loop during external sort Key: SPARK-8202 URL: https://issues.apache.org/jira/browse/SPARK-8202 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 1.4.0 Reporter: Davies Liu Assignee: Davies Liu Priority: Critical
The batch size during external sort will grow up to max 10000, then shrink down to zero, causing infinite loop. Given the assumption that the items usually have similar size, so we don't need to adjust the batch size after first spill. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org