[ https://issues.apache.org/jira/browse/SPARK-7998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin updated SPARK-7998: ------------------------------- Description: The current freqItems API is really awkward to use. It returns a DataFrame with a single row, in which each value is an array of frequent items. This design doesn't work well for exploratory data analysis (running show -- when there are more than 2 or 3 frequent values, the values get cut off): {code} In [74]: df.stat.freqItems(["a", "b", "c"], 0.4).show() +------------------+------------------+-----------------+ | a_freqItems| b_freqItems| c_freqItems| +------------------+------------------+-----------------+ |ArrayBuffer(11, 1)|ArrayBuffer(2, 22)|ArrayBuffer(1, 3)| +------------------+------------------+-----------------+ {code} It also doesn't work well for serious engineering, since it is hard to get the value out. We should create a new function (so we maintain source/binary compatibility) that returns a list of list of values. was: The current freqItems API is really awkward to use. It returns a DataFrame with a single row, in which each value is an array of frequent items. This design doesn't work well for exploratory data analysis (running show -- when there are more than 2 or 3 frequent values, the values get cut off): {code} In [74]: df.stat.freqItems(["a", "b", "c"], 0.4).show() +------------------+------------------+-----------------+ | a_freqItems| b_freqItems| c_freqItems| +------------------+------------------+-----------------+ |ArrayBuffer(11, 1)|ArrayBuffer(2, 22)|ArrayBuffer(1, 3)| +------------------+------------------+-----------------+ {code} It also doesn't work well for serious engineering, since it is hard to get the value out. We should just create a new function (so we maintain source/binary compatibility) that returns a list of list of values. > A better frequent item API > -------------------------- > > Key: SPARK-7998 > URL: https://issues.apache.org/jira/browse/SPARK-7998 > Project: Spark > Issue Type: Sub-task > Components: SQL > Reporter: Reynold Xin > > The current freqItems API is really awkward to use. It returns a DataFrame > with a single row, in which each value is an array of frequent items. > This design doesn't work well for exploratory data analysis (running show -- > when there are more than 2 or 3 frequent values, the values get cut off): > {code} > In [74]: df.stat.freqItems(["a", "b", "c"], 0.4).show() > +------------------+------------------+-----------------+ > | a_freqItems| b_freqItems| c_freqItems| > +------------------+------------------+-----------------+ > |ArrayBuffer(11, 1)|ArrayBuffer(2, 22)|ArrayBuffer(1, 3)| > +------------------+------------------+-----------------+ > {code} > It also doesn't work well for serious engineering, since it is hard to get > the value out. > We should create a new function (so we maintain source/binary compatibility) > that returns a list of list of values. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org