[
https://issues.apache.org/jira/browse/HBASE-24901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180796#comment-17180796
]
Elliot Miller commented on HBASE-24901:
---------------------------------------
Another benefit of this patch is that it updates how we output the results of a
scan or get. This patch takes advantage of {{ResultScanner}}'s iterator and
{{CellScanner}}'s current/advance to stream the output without collecting all
the rows first.
Running a local test on my laptop, I scanned a 1G table with the JVM heap
limited to 120M and the whole table printed! To be sure, I redirected the shell
output to a file and the file was {{1008M}}. I verified the heap limit with
jmap. My _limited_ testing so far indicates that you should be able to scan
any(ish) size table without running out of memory.
> Create versatile hbase-shell table formatter
> --------------------------------------------
>
> Key: HBASE-24901
> URL: https://issues.apache.org/jira/browse/HBASE-24901
> Project: HBase
> Issue Type: Improvement
> Components: shell
> Affects Versions: 3.0.0-alpha-1
> Reporter: Elliot Miller
> Assignee: Elliot Miller
> Priority: Major
> Attachments: HBASE-24901_scan_output_comparison.png
>
>
> As a user, I would like a simple interface for shell output that can be
> expressed as a table (ie. output with a fixed number of columns and
> potentially many rows). To be clear, this new formatter is not specifically
> for HBase "tables." Table is used in the broader sense here.
> Goals
> - Do not require more than one output cell loaded in memory at a time
> - Support many implementations like aligned human-friendly tables, unaligned
> delimited, and JSON
> Non-goals
> - Don't load all the headers into memory at once.
> - This may seem like a goal with merit, but we are unlikely to find a use
> case for this formatter with many columns. For example: since HBase tables
> aren't relational, our scan output will not have an output column for every
> HBase column. Instead, each output row will correspond to an HBase cell.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
