[ https://issues.apache.org/jira/browse/LENS-910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093417#comment-15093417 ]
Hadoop QA commented on LENS-910: -------------------------------- Applied patch: [LENS-910-5.patch|https://issues.apache.org/jira/secure/attachment/12781745/LENS-910-5.patch] and ran command: mvn clean install -fae. Result: Success. Build Job: https://builds.apache.org/job/PreCommit-Lens-Build/248/ > Add session config to skip filtering cube related tables from all the tables > in a database > ------------------------------------------------------------------------------------------ > > Key: LENS-910 > URL: https://issues.apache.org/jira/browse/LENS-910 > Project: Apache Lens > Issue Type: Improvement > Reporter: Deepak Barr > Assignee: Deepak Barr > Priority: Minor > Attachments: LENS-910-5.patch > > > {code:title=CubeMetastoreServiceImpl.java|borderStyle=solid} > private List<String> getNativeTablesFromDB(LensSessionHandle sessionid, > String dbName, boolean prependDbName) > throws LensException { > IMetaStoreClient msc = null; > try { > msc = getSession(sessionid).getMetaStoreClient(); > List<String> tables = msc.getAllTables( > dbName); > List<String> result = new ArrayList<String>(); > if (tables != null && !tables.isEmpty()) { > List<org.apache.hadoop.hive.metastore.api.Table> tblObjects = > msc.getTableObjectsByName(dbName, tables); > Iterator<org.apache.hadoop.hive.metastore.api.Table> it = > tblObjects.iterator(); > while (it.hasNext()) { > org.apache.hadoop.hive.metastore.api.Table tbl = it.next(); > if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == > null) { > if (prependDbName) { > result.add(dbName + "." + tbl.getTableName()); > } else { > result.add(tbl.getTableName()); > } > } > } > } > return result; > } catch (Exception e) { > throw new LensException("Error getting native tables from DB", e); > } > } > {code} > We have approx. 18K tables in one of our hive databases. When fetching native > tables, the getNativeTablesFromDB() filters out the cube related tables from > the superset using table property(cube.table.type). This filtering is taking > a long time for our heavy databases. The call to metastore API > [msc.getTableObjectsByName()] is taking most of the time. Since, the cube > related tables may not be present in some/most databases, the filtering is > not necessary for them. I think we can add a session property, so that a user > can skip filtering if he wants to. Thoughts ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)