[ https://issues.apache.org/jira/browse/LENS-910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093486#comment-15093486 ]
Hudson commented on LENS-910: ----------------------------- UNSTABLE: Integrated in Lens-Commit-Java8 #84 (See [https://builds.apache.org/job/Lens-Commit-Java8/84/]) LENS-910 : Add session config to skip filtering cube related tables from (deepakbarr: rev edcdd968541b6de572b65abbe92b4f3601c2f7a3) * lens-server/src/main/resources/lenssession-default.xml * lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java * lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java * lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java * src/site/apt/admin/session-config.apt * lens-cli/src/test/java/org/apache/lens/cli/TestLensNativeTableCommands.java * lens-server/src/test/java/org/apache/lens/server/metastore/TestMetastoreService.java > Add session config to skip filtering cube related tables from all the tables > in a database > ------------------------------------------------------------------------------------------ > > Key: LENS-910 > URL: https://issues.apache.org/jira/browse/LENS-910 > Project: Apache Lens > Issue Type: Improvement > Reporter: Deepak Barr > Assignee: Deepak Barr > Priority: Minor > Attachments: LENS-910-5.patch > > > {code:title=CubeMetastoreServiceImpl.java|borderStyle=solid} > private List<String> getNativeTablesFromDB(LensSessionHandle sessionid, > String dbName, boolean prependDbName) > throws LensException { > IMetaStoreClient msc = null; > try { > msc = getSession(sessionid).getMetaStoreClient(); > List<String> tables = msc.getAllTables( > dbName); > List<String> result = new ArrayList<String>(); > if (tables != null && !tables.isEmpty()) { > List<org.apache.hadoop.hive.metastore.api.Table> tblObjects = > msc.getTableObjectsByName(dbName, tables); > Iterator<org.apache.hadoop.hive.metastore.api.Table> it = > tblObjects.iterator(); > while (it.hasNext()) { > org.apache.hadoop.hive.metastore.api.Table tbl = it.next(); > if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == > null) { > if (prependDbName) { > result.add(dbName + "." + tbl.getTableName()); > } else { > result.add(tbl.getTableName()); > } > } > } > } > return result; > } catch (Exception e) { > throw new LensException("Error getting native tables from DB", e); > } > } > {code} > We have approx. 18K tables in one of our hive databases. When fetching native > tables, the getNativeTablesFromDB() filters out the cube related tables from > the superset using table property(cube.table.type). This filtering is taking > a long time for our heavy databases. The call to metastore API > [msc.getTableObjectsByName()] is taking most of the time. Since, the cube > related tables may not be present in some/most databases, the filtering is > not necessary for them. I think we can add a session property, so that a user > can skip filtering if he wants to. Thoughts ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)