----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70512/#review215013 -----------------------------------------------------------
addons/impala-bridge-shim/pom.xml Lines 28 (patched) <https://reviews.apache.org/r/70512/#comment301354> shim is not needed; since this will be used as a standalone utility. addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java Lines 35 (patched) <https://reviews.apache.org/r/70512/#comment301320> TEMP_TABLE_PREFIX is never used; consider removing it. addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java Lines 65 (patched) <https://reviews.apache.org/r/70512/#comment301333> return after LOG.warn stmt; might hit NPE in line #73 addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java Lines 33 (patched) <https://reviews.apache.org/r/70512/#comment301336> why the following queries are not handled? * CREATE TABLE * INSERT INTO TABLE <table_name> SELECT * INSERT OVERWRITE TABLE addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java Lines 459 (patched) <https://reviews.apache.org/r/70512/#comment301325> columnPosition never used? consider removing it. addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java Lines 490 (patched) <https://reviews.apache.org/r/70512/#comment301324> can we have new model change to create "impala_process" typedef with supertype "process" as part of this patch? Will be good to see impala_process connecting hive entities in the server side. addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java Lines 71 (patched) <https://reviews.apache.org/r/70512/#comment301337> what cases do we want to skip impala_process creation? addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java Lines 133 (patched) <https://reviews.apache.org/r/70512/#comment301343> remove edges==null check; CollectionUtils.isEmpty(edges) does null check as well. addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java Lines 138 (patched) <https://reviews.apache.org/r/70512/#comment301346> lineageInputsCount is unused; consider removing if not used. addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java Lines 204 (patched) <https://reviews.apache.org/r/70512/#comment301353> dependencyType can not always be be SIMPLE; https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageInfo.java#L59 addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java Lines 26 (patched) <https://reviews.apache.org/r/70512/#comment301321> consider removing explicit "private" modifier. addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java Lines 38 (patched) <https://reviews.apache.org/r/70512/#comment301352> make edgeType as enum addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java Lines 36 (patched) <https://reviews.apache.org/r/70512/#comment301322> consider changing "id" type from Integer to Long. addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java Lines 39 (patched) <https://reviews.apache.org/r/70512/#comment301351> make vertexType as enum - Sarath Subramanian On May 2, 2019, 1:31 p.m., Na Li wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70512/ > ----------------------------------------------------------- > > (Updated May 2, 2019, 1:31 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath > Subramanian, and Xinran Tinney. > > > Repository: atlas > > > Description > ------- > > Impala generates lineage records for its commands. This new feature will read > Impala lineage file, convert the lineage record to Atlas entities and send > them to Atlas. In this way, Atlas can get lineage of Impala operation. > > The metadata referred in the lineage are captured in Hive Metastore hook and > sent to Atlas. This work is done in ATLAS-3148 > > This jira only supports the Impala command "create view". Following jira will > add support for more Impala commands. > > > Diffs > ----- > > addons/impala-bridge-shim/pom.xml PRE-CREATION > > addons/impala-bridge-shim/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > addons/impala-bridge/pom.xml PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataTypes.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageQuery.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java > PRE-CREATION > addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-application.properties > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION > addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION > addons/impala-bridge/src/test/resources/users-credentials.properties > PRE-CREATION > pom.xml 7de5d31 > > > Diff: https://reviews.apache.org/r/70512/diff/13/ > > > Testing > ------- > > Run the tool in real cluster that has Atlas server with Impala lineage file > as input for creating view. The Atlas UI displays hive_lineage lineage and > hive_column_lineage. > Add new integration tests and they pass > > > Thanks, > > Na Li > >
