[ https://issues.apache.org/jira/browse/HBASE-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Tokhy updated HBASE-4435: ------------------------------- Attachment: HBASE-4435-v2.patch I have a newer version of the patch: Improvements: 1) Added implementations of ColumnInterpreter classes so both AggregationClient and GroupByClient could perform aggregations on Long, Short, Integer, Double, Float, Character (or unsigned short), and BigDecimal types. 2) The GroupByStatsValues class is a Java generic that constrains on Java types that only implement the 'Number' interface. This way the generic is constrained for those types at compile time. 3) Previously, a HashMap was returned at the end of each RPC call. HashMap uses java.io.Serializable, which is relatively heavyweight. Switched to using the Hadoop Writable interface so all objects passed between clients and regionservers use the Hadoop Writable interface. 4) Fixed some validateParameter bugs in the previous patch which would allow selections of column qualifiers not found in the Scan object to go through. Caveats: 1) This works well if your resultset fits into memory as group by values are aggregated into a HashMap on the client. Therefore, if the cardinality of the aggregation table is too high, you may get an OOME. 2) All aggregations are calculated by the 'GroupByStatsValues' container. Perhaps at object construction, a 'statsvalues' can be constructed to only perform some of the aggregations instead of all of them at the same time. However this operation is Scan (IO) bound, so improvements would be minimal here. 3) Like all coprocessors that accept a Scan object, if the aggregation is performing a full table scan, this will run on all regionservers. Each region level coprocessor is loaded into an IPC handler (default of 10) on the regionserver. If the regionserver has more regions than IPC handlers, only 10 group by operations will run at a time. Depending on your table schema, region size and blockCacheHitRatio, your mileage may vary. If data can be preaggregated for a group by operation, this patch would be handy for aggregating a single column value projection of the original full table. A column oriented representation of the original table would work well in this case, or possibly a client/coprocessor managed secondary index. The patch applies cleanly onto HBase 0.92.1 and HBase 0.94.1. > Add Group By functionality using Coprocessors > --------------------------------------------- > > Key: HBASE-4435 > URL: https://issues.apache.org/jira/browse/HBASE-4435 > Project: HBase > Issue Type: Improvement > Components: Coprocessors > Reporter: Nichole Treadway > Priority: Minor > Attachments: HBase-4435.patch, HBASE-4435-v2.patch > > > Adds in a Group By -like functionality to HBase, using the Coprocessor > framework. > It provides the ability to group the result set on one or more columns > (groupBy families). It computes statistics (max, min, sum, count, sum of > squares, number missing) for a second column, called the stats column. > To use, I've provided two implementations. > 1. In the first, you specify a single group-by column and a stats field: > statsMap = gbc.getStats(tableName, scan, groupByFamily, > groupByQualifier, statsFamily, statsQualifier, statsFieldColumnInterpreter); > The result is a map with the Group By column value (as a String) to a > GroupByStatsValues object. The GroupByStatsValues object has max,min,sum etc. > of the stats column for that group. > 2. The second implementation allows you to specify a list of group-by columns > and a stats field. The List of group-by columns is expected to contain lists > of {column family, qualifier} pairs. > statsMap = gbc.getStats(tableName, scan, listOfGroupByColumns, > statsFamily, statsQualifier, statsFieldColumnInterpreter); > The GroupByStatsValues code is adapted from the Solr Stats component. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira