Github user fhueske commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/172#discussion_r19728981
--- Diff:
flink-addons/flink-hbase/src/main/java/org/apache/flink/addons/hbase/TableInputFormat.java
---
@@ -310,17 +264,13 @@ public void open(TableInputSplit split) throws
IOException {
@Override
public TableInputSplit[] createInputSplits(final int minNumSplits)
throws IOException {
+ HTable table = createTable(parameters);
- if (this.table == null) {
- throw new IOException("No table was provided.");
- }
-
- final Pair<byte[][], byte[][]> keys =
this.table.getStartEndKeys();
-
+ final Pair<byte[][], byte[][]> keys = table.getStartEndKeys();
if (keys == null || keys.getFirst() == null ||
keys.getFirst().length == 0) {
-
throw new IOException("Expecting at least one region.");
}
+ Scan scan = createScanner(parameters);
int count = 0;
final List<TableInputSplit> splits = new
ArrayList<TableInputSplit>(keys.getFirst().length);
--- End diff --
If there is a huge table with a several thousand regions, the initial size
of the split list would be that big even if only a few regions should be read.
If we start with a smaller value, the list would grow by need (with a bit of
overhead for reorganization of cause).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
