Juliusz Sompolski created SPARK-24013:
-----------------------------------------
Summary: ApproximatePercentile grinds to a halt on sorted input.
Key: SPARK-24013
URL: https://issues.apache.org/jira/browse/SPARK-24013
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.0
Reporter: Juliusz Sompolski
Running
{code}
sql("select approx_percentile(rid, array(0.1)) from (select rand() as rid from
range(10000000))").collect()
{code}
takes 7 seconds, while
{code}
sql("select approx_percentile(id, array(0.1)) from range(10000000)").collect()
{code}
grinds to a halt - processes the first million rows quickly, and then slows
down to a few thousands rows / second (4m rows processed after 20 minutes).
Thread dumps show that it spends time in QuantileSummary.compress.
Seems it hits some edge case inefficiency when dealing with sorted data?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
