> On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > > Lines 64 (patched) > > <https://reviews.apache.org/r/70512/diff/14/?file=2142926#file2142926line64> > > > > Replace with > > LOG.info("Importing: {}", filename);
done > On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > > Lines 65 (patched) > > <https://reviews.apache.org/r/70512/diff/14/?file=2142926#file2142926line65> > > > > What are we doing with the return value of this method? nothing. I change the return to void since the list is processed inside this function. > On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > > Lines 118 (patched) > > <https://reviews.apache.org/r/70512/diff/14/?file=2142931#file2142931line118> > > > > What if node itself is null? add code to throw exception in this case. > On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > > Lines 158 (patched) > > <https://reviews.apache.org/r/70512/diff/14/?file=2142932#file2142932line158> > > > > Do we need this check? We can just put LOG.debug. If log level is set > > to debug it will log it else it will not isnt it? remove the check. The check is necessary for performance reason if it takes a lot of effort to construct the log message. It is not the case here, so remove the check. > On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > > Lines 165 (patched) > > <https://reviews.apache.org/r/70512/diff/14/?file=2142932#file2142932line165> > > > > Remove extra line done > On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > > Lines 171 (patched) > > <https://reviews.apache.org/r/70512/diff/14/?file=2142932#file2142932line171> > > > > Remove extra line done > On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > > Lines 189 (patched) > > <https://reviews.apache.org/r/70512/diff/14/?file=2142932#file2142932line189> > > > > Remove extra line done > On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > > Lines 203 (patched) > > <https://reviews.apache.org/r/70512/diff/14/?file=2142932#file2142932line203> > > > > Change ":" to constant done > On May 7, 2019, 2:47 a.m., Aadarsh Jajodia wrote: > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > > Lines 227 (patched) > > <https://reviews.apache.org/r/70512/diff/14/?file=2142932#file2142932line227> > > > > Do we need the isDebugEnabled check? We can just put LOG.debug. If log > > level is set to debug it will log it else it will not isnt it? > > > > Do we also need the isColumnLineage check? I see that above we are > > creating objects of columnlineage and hence they will never be NULL. I feel > > this check can be avoided here updated - Na ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70512/#review215078 ----------------------------------------------------------- On May 4, 2019, 9:23 p.m., Na Li wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70512/ > ----------------------------------------------------------- > > (Updated May 4, 2019, 9:23 p.m.) > > > Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, madhan, Sarath > Subramanian, and Xinran Tinney. > > > Repository: atlas > > > Description > ------- > > Impala generates lineage records for its commands. This new feature will read > Impala lineage file, convert the lineage record to Atlas entities and send > them to Atlas. In this way, Atlas can get lineage of Impala operation. > > The metadata referred in the lineage are captured in Hive Metastore hook and > sent to Atlas. This work is done in ATLAS-3148 > > This jira only supports the Impala command "create view". Following jira will > add support for more Impala commands. > > > Diffs > ----- > > addons/impala-bridge/pom.xml PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java > PRE-CREATION > > addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java > PRE-CREATION > addons/impala-bridge/src/main/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/main/resources/import-impala.sh PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java > PRE-CREATION > > addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-application.properties > PRE-CREATION > addons/impala-bridge/src/test/resources/atlas-log4j.xml PRE-CREATION > addons/impala-bridge/src/test/resources/hive-site.xml PRE-CREATION > addons/impala-bridge/src/test/resources/impala1.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala2.json PRE-CREATION > addons/impala-bridge/src/test/resources/impala3.json PRE-CREATION > addons/impala-bridge/src/test/resources/users-credentials.properties > PRE-CREATION > addons/models/1000-Hadoop/1090-impala_model.json PRE-CREATION > pom.xml 7de5d31 > > > Diff: https://reviews.apache.org/r/70512/diff/14/ > > > Testing > ------- > > Run the tool in real cluster that has Atlas server with Impala lineage file > as input for creating view. The Atlas UI displays hive_lineage lineage and > hive_column_lineage. > Add new integration tests and they pass > > > Thanks, > > Na Li > >
