Deepak Barr created LENS-796:
--------------------------------
Summary: Slow response times for /metastore/nativetables API
Key: LENS-796
URL: https://issues.apache.org/jira/browse/LENS-796
Project: Apache Lens
Issue Type: Improvement
Reporter: Deepak Barr
Assignee: Deepak Barr
To obtain list of native tables, CubeMetastoreService does the following -
1. Fetches the list of tables ( one MetastoreClient call)
2. Filters out the cube tables from the list. The filtering happens by looking
at the table properties from the Table object. This table object is obtained
with another Metastore call. So, If there are 'n' tables, there will be 'n'
metastore calls.
Here is the code snippet :
private List<String> getTablesFromDB(LensSessionHandle sessionid,
String dbName, boolean prependDbName)
throws MetaException, UnknownDBException, HiveSQLException, TException,
LensException {
List<String> tables =
getSession(sessionid).getMetaStoreClient().getAllTables(
dbName);
List<String> result = new ArrayList<String>();
if (tables != null && !tables.isEmpty()) {
Iterator<String> it = tables.iterator();
while (it.hasNext()) {
String tblName = it.next();
org.apache.hadoop.hive.metastore.api.Table tbl =
getSession(sessionid).getMetaStoreClient().getTable(dbName, tblName);
if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == null)
{
if (prependDbName) {
result.add(dbName + "." + tblName);
} else {
result.add(tblName);
}
}
}
}
return result;
}
Instead of this, we can directly fetch the list of table objects for our list
of table names in a single API call using
getMetaStoreClient().getTableObjectsByName() method.
Currently, one of our databases contain 8000+ tables which leads to very very
long response times.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
