Reynold Xin created SPARK-7998:
----------------------------------
Summary: A better frequent item API
Key: SPARK-7998
URL: https://issues.apache.org/jira/browse/SPARK-7998
Project: Spark
Issue Type: Sub-task
Components: SQL
Reporter: Reynold Xin
The current freqItems API is really awkward to use. It returns a DataFrame with
a single row, in which each value is an array of frequent items.
This design doesn't work well for exploratory data analysis (running show --
when there are more than 2 or 3 frequent values, the values get cut off):
{code}
In [74]: df.stat.freqItems(["a", "b", "c"], 0.4).show()
+------------------+------------------+-----------------+
| a_freqItems| b_freqItems| c_freqItems|
+------------------+------------------+-----------------+
|ArrayBuffer(11, 1)|ArrayBuffer(2, 22)|ArrayBuffer(1, 3)|
+------------------+------------------+-----------------+
{code}
It also doesn't work well for serious engineering, since it is hard to get the
value out.
We should just create a new function (so we maintain source/binary
compatibility) that returns a list of list of values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
