Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-22 Thread Sridhar K 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215475
---


Ship it!




Ship It!

- Sridhar K


On May 22, 2019, 5:36 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 22, 2019, 5:36 p.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  cc62955 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  6e4d321 
>   addons/impala-bridge/src/test/resources/impala1.json 8f747f6 
>   addons/impala-bridge/src/test/resources/impala2.json 239797b 
>   addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
>   addons/impala-bridge/src/test/resources/impala5.json 854969b 
>   addons/impala-bridge/src/test/resources/impala6.json f136180 
>   addons/impala-bridge/src/test/resources/impala7.json f9ee670 
>   addons/impala-bridge/src/test/resources/impalaAlterViewAsSelect.json 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaCreateTableAsSelect.json 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaCreateView.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaCreateViewNoCreateTime.json 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaInsertIntoAsSelect.json 
> PRE-CREATION 
>   addons/impala-hook-api/pom.xml PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
>  PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java
>  PRE-CREATION 
>   pom.xml 0a8f9b7 
> 
> 
> Diff: https://reviews.apache.org/r/70619/diff/11/
> 
> 
> Testing
> ---
> 
> add integration tests 
> ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
> situation that there is no createTime in lineage record
> change the file content for other integration tests in ImpalaLineageToolIT to 
> handle new lineage record format
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-22 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215472
---


Ship it!




Ship It!

- Sarath Subramanian


On May 22, 2019, 10:36 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 22, 2019, 10:36 a.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  cc62955 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  6e4d321 
>   addons/impala-bridge/src/test/resources/impala1.json 8f747f6 
>   addons/impala-bridge/src/test/resources/impala2.json 239797b 
>   addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
>   addons/impala-bridge/src/test/resources/impala5.json 854969b 
>   addons/impala-bridge/src/test/resources/impala6.json f136180 
>   addons/impala-bridge/src/test/resources/impala7.json f9ee670 
>   addons/impala-bridge/src/test/resources/impalaAlterViewAsSelect.json 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaCreateTableAsSelect.json 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaCreateView.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaCreateViewNoCreateTime.json 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaInsertIntoAsSelect.json 
> PRE-CREATION 
>   addons/impala-hook-api/pom.xml PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
>  PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java
>  PRE-CREATION 
>   pom.xml 0a8f9b7 
> 
> 
> Diff: https://reviews.apache.org/r/70619/diff/11/
> 
> 
> Testing
> ---
> 
> add integration tests 
> ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
> situation that there is no createTime in lineage record
> change the file content for other integration tests in ImpalaLineageToolIT to 
> handle new lineage record format
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-22 Thread Na Li via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/
---

(Updated May 22, 2019, 5:36 p.m.)


Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

Integrate with Impala hook using Impala private jar with post exec hook

changes:

1) handle changed lineage record format change
The lineage record format has changed from having seperate table vertex 

  {  
 "id":4,
 "vertexType":"TABLE",
 "vertexId":"db_1.table_1",
 "createTime":155475
  }
  
To embed the table create time inside column vertex

{
  "id": 3,
  "vertexType": "COLUMN",
  "vertexId": "db_1.table_1.tinyint_col",
  "metadata": {
"createTime": 1557860978,
"tableName": "db_1.table_1"
  }
}

2) add Impala hook API code to allow Atlas integration code to compile without 
Impala updates


Diffs (updated)
-

  addons/impala-bridge-shim/pom.xml PRE-CREATION 
  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  addons/impala-bridge/pom.xml 8e7e952 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java 
PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 fdb6748 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 afe296c 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 e4a38a3 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 7c1103a 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 82672c9 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 cc62955 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 6e4d321 
  addons/impala-bridge/src/test/resources/impala1.json 8f747f6 
  addons/impala-bridge/src/test/resources/impala2.json 239797b 
  addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
  addons/impala-bridge/src/test/resources/impala5.json 854969b 
  addons/impala-bridge/src/test/resources/impala6.json f136180 
  addons/impala-bridge/src/test/resources/impala7.json f9ee670 
  addons/impala-bridge/src/test/resources/impalaAlterViewAsSelect.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateTableAsSelect.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateView.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateViewNoCreateTime.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaInsertIntoAsSelect.json 
PRE-CREATION 
  addons/impala-hook-api/pom.xml PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java 
PRE-CREATION 
  pom.xml 0a8f9b7 


Diff: https://reviews.apache.org/r/70619/diff/11/

Changes: https://reviews.apache.org/r/70619/diff/10-11/


Testing
---

add integration tests 
ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
situation that there is no createTime in lineage record
change the file content for other integration tests in ImpalaLineageToolIT to 
handle new lineage record format


Thanks,

Na Li

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-22 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215450
---


Fix it, then Ship it!





addons/impala-bridge/pom.xml
Line 184 (original), 183 (patched)


do we need this hive library as dependency? also review other hive 
dependencies in line# 196, 217 and consider removing it.



addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
Line 69 (original), 68 (patched)


add null check for 'lineageQuery'



addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
Lines 274 (patched)


duplicate code in lines 274-279 and 323-328; consider refactoring to a 
method - createTableNode(Long createTime);



addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
Lines 283 (patched)


consider removing void return statement.


- Sarath Subramanian


On May 21, 2019, 7:35 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 21, 2019, 7:35 p.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  cc62955 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  6e4d321 
>   addons/impala-bridge/src/test/resources/impala1.json 8f747f6 
>   addons/impala-bridge/src/test/resources/impala2.json 239797b 
>   addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
>   addons/impala-bridge/src/test/resources/impala5.json 854969b 
>   addons/impala-bridge/src/test/resources/impala6.json f136180 
>   addons/impala-bridge/src/test/resources/impala7.json f9ee670 
>   addons/impala-bridge/src/test/resources/impalaAlterViewAsSelect.json 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaCreateTableAsSelect.json 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaCreateView.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaCreateViewNoCreateTime.json 
> PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impalaInsertIntoAsSelect.json 
> PRE-CREATION 
>   addons/impala-hook-api/pom.xml PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
>  PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java
>  PRE-CREATION 
>   pom.xml 98c2d69 
> 
> 
> Diff: https://reviews.apache.org/r/70619/diff/10/
> 
> 
> Testing
> ---
> 
> add integration tests 
> ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
> situation

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-21 Thread Na Li via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/
---

(Updated May 22, 2019, 2:35 a.m.)


Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

Integrate with Impala hook using Impala private jar with post exec hook

changes:

1) handle changed lineage record format change
The lineage record format has changed from having seperate table vertex 

  {  
 "id":4,
 "vertexType":"TABLE",
 "vertexId":"db_1.table_1",
 "createTime":155475
  }
  
To embed the table create time inside column vertex

{
  "id": 3,
  "vertexType": "COLUMN",
  "vertexId": "db_1.table_1.tinyint_col",
  "metadata": {
"createTime": 1557860978,
"tableName": "db_1.table_1"
  }
}

2) add Impala hook API code to allow Atlas integration code to compile without 
Impala updates


Diffs (updated)
-

  addons/impala-bridge-shim/pom.xml PRE-CREATION 
  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  addons/impala-bridge/pom.xml 8e7e952 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java 
PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 fdb6748 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 afe296c 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 e4a38a3 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 7c1103a 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 82672c9 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 cc62955 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 6e4d321 
  addons/impala-bridge/src/test/resources/impala1.json 8f747f6 
  addons/impala-bridge/src/test/resources/impala2.json 239797b 
  addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
  addons/impala-bridge/src/test/resources/impala5.json 854969b 
  addons/impala-bridge/src/test/resources/impala6.json f136180 
  addons/impala-bridge/src/test/resources/impala7.json f9ee670 
  addons/impala-bridge/src/test/resources/impalaAlterViewAsSelect.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateTableAsSelect.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateView.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaCreateViewNoCreateTime.json 
PRE-CREATION 
  addons/impala-bridge/src/test/resources/impalaInsertIntoAsSelect.json 
PRE-CREATION 
  addons/impala-hook-api/pom.xml PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java 
PRE-CREATION 
  pom.xml 98c2d69 


Diff: https://reviews.apache.org/r/70619/diff/10/

Changes: https://reviews.apache.org/r/70619/diff/9-10/


Testing
---

add integration tests 
ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
situation that there is no createTime in lineage record
change the file content for other integration tests in ImpalaLineageToolIT to 
handle new lineage record format


Thanks,

Na Li

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-21 Thread Sarath Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215432
---




addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
Lines 283 (patched)


remove void return statement.


- Sarath Subramanian


On May 21, 2019, 7:15 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 21, 2019, 7:15 p.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  7c9abc8 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  88faace 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
>  b9d6cbb 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  b9dd894 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
>  10ce448 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
>  892ee9b 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  a3ddf53 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  a893b88 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
>  27bdc72 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
>  8ec3f85 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  251507e 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDependencyType.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaNode.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaOperationType.java
>  PRE-CREATION 
>   
>

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-21 Thread Na Li via Review Board 


> On May 21, 2019, 7:09 p.m., Sridhar K wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
> > Lines 19 (patched)
> > 
> >
> > Please package name w.r.t. directories. There are lot of classes in 
> > wrong folder strucutre. Please fix it.
> 
> Na Li wrote:
> will be fixed in https://issues.apache.org/jira/browse/ATLAS-3221

I have fixed the package issue in review version 9. However, do you really want 
to have integration change and fixing packaging in the same patch? As you can 
see, so many changes, the review gets really big, and you don't what has 
changed for integration with Impala.

That is preciously why I want to focus this issue ATLAS-3080 on integration, 
and ATLAS-3221 on fixing packaging issue.


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215417
---


On May 22, 2019, 2:15 a.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 22, 2019, 2:15 a.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
>  7c9abc8 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
>  88faace 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
>  b9d6cbb 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
>  b9dd894 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
>  10ce448 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
>  892ee9b 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
>  a3ddf53 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
>  a893b88 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
>  27bdc72 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
>  8ec3f85 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
>  251507e 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java
>  PRE-CREATION 
>   
>

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-21 Thread Na Li via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/
---

(Updated May 22, 2019, 2:15 a.m.)


Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

Integrate with Impala hook using Impala private jar with post exec hook

changes:

1) handle changed lineage record format change
The lineage record format has changed from having seperate table vertex 

  {  
 "id":4,
 "vertexType":"TABLE",
 "vertexId":"db_1.table_1",
 "createTime":155475
  }
  
To embed the table create time inside column vertex

{
  "id": 3,
  "vertexType": "COLUMN",
  "vertexId": "db_1.table_1.tinyint_col",
  "metadata": {
"createTime": 1557860978,
"tableName": "db_1.table_1"
  }
}

2) add Impala hook API code to allow Atlas integration code to compile without 
Impala updates


Diffs (updated)
-

  addons/impala-bridge-shim/pom.xml PRE-CREATION 
  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  addons/impala-bridge/pom.xml 8e7e952 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/ImpalaLineageTool.java
 7c9abc8 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/AtlasImpalaHookContext.java
 88faace 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaIdentifierParser.java
 b9d6cbb 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 fdb6748 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaOperationParser.java
 b9dd894 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 afe296c 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 e4a38a3 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 7c1103a 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDataType.java
 10ce448 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaDependencyType.java
 892ee9b 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaNode.java
 a3ddf53 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaOperationType.java
 a893b88 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaQuery.java
 27bdc72 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/ImpalaVertexType.java
 8ec3f85 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageEdge.java
 251507e 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 82672c9 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/ImpalaLineageTool.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaHook.java 
PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaIdentifierParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaOperationParser.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/CreateImpalaProcess.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDataType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaDependencyType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaNode.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaOperationType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaQuery.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/ImpalaVertexType.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageEdge.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertex.java
 PRE-CREATION 
  
addons/impala-bridge/src/main/java/org/apache/atlas/impala/model/LineageVertexMetadata.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 cc62955 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 6e4d321 
  addons/impala-bridge/src/test/resources/impala1.json 8f747f6

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-21 Thread Na Li via Review Board 


> On May 21, 2019, 7:27 p.m., Sridhar K wrote:
> > addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
> > Lines 107 (patched)
> > 
> >
> > getTableNameFromColumn ==> getTableNameFromVertex
> 
> Na Li wrote:
> This is function to get table name from column vertext
> 
> Na Li wrote:
> public String getTableNameFromColumn(LineageVertex vertex) {
> if (vertex.getVertexType() == ImpalaVertexType.COLUMN) {
> LineageVertexMetadata metadata = vertex.getMetadata();
> 
> if (metadata != null) {
> return metadata.getTableName();
> }
> }
> 
> return getTableNameFromColumn(vertex.getVertexId());
> }

changed


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215422
---


On May 21, 2019, 10:10 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 21, 2019, 10:10 p.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  cc62955 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  6e4d321 
>   addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
>   addons/impala-bridge/src/test/resources/impala4.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala5.json 854969b 
>   addons/impala-bridge/src/test/resources/impala6.json f136180 
>   addons/impala-bridge/src/test/resources/impala7.json f9ee670 
>   addons/impala-hook-api/pom.xml PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
>  PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java
>  PRE-CREATION 
>   pom.xml 98c2d69 
> 
> 
> Diff: https://reviews.apache.org/r/70619/diff/8/
> 
> 
> Testing
> ---
> 
> add integration tests 
> ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
> situation that there is no createTime in lineage record
> change the file content for other integration tests in ImpalaLineageToolIT to 
> handle new lineage record format
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-21 Thread Na Li via Review Board 


> On May 21, 2019, 6:18 p.m., Sridhar K wrote:
> > addons/impala-bridge/pom.xml
> > Lines 59 (patched)
> > 
> >
> > Can we remove the project version hereand define dependency version 
> > in root pom.xml file's dependency management section.
> 
> Na Li wrote:
> No. I have to put version here to avoid compilation error (I don't know 
> why it is needed). "${project.version}" is the version defined in root 
> pom.xml at https://github.com/apache/atlas/blob/master/pom.xml#L32.
> 
> hive-bridge/pom.xml uses that too. 
> https://github.com/apache/atlas/blob/master/addons/hive-bridge/pom.xml#L123

done


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215416
---


On May 21, 2019, 10:10 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 21, 2019, 10:10 p.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  cc62955 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  6e4d321 
>   addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
>   addons/impala-bridge/src/test/resources/impala4.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala5.json 854969b 
>   addons/impala-bridge/src/test/resources/impala6.json f136180 
>   addons/impala-bridge/src/test/resources/impala7.json f9ee670 
>   addons/impala-hook-api/pom.xml PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
>  PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java
>  PRE-CREATION 
>   pom.xml 98c2d69 
> 
> 
> Diff: https://reviews.apache.org/r/70619/diff/8/
> 
> 
> Testing
> ---
> 
> add integration tests 
> ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
> situation that there is no createTime in lineage record
> change the file content for other integration tests in ImpalaLineageToolIT to 
> handle new lineage record format
> 
> 
> Thanks,
> 
> Na Li
> 
>

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-21 Thread Na Li via Review Board 


> On May 21, 2019, 6:18 p.m., Sridhar K wrote:
> > addons/impala-bridge-shim/pom.xml
> > Lines 41 (patched)
> > 
> >
> > Can we remove the project version hereand define dependency version 
> > in root pom.xml file's dependency management section.
> 
> Na Li wrote:
> I tried to not put the version here, but did not work. Compilation failed 
> in dependency for not specifying version. 
> 
> 
> without this line "${project.version}", I got the 
> following error.
> 
> [ERROR]   The project org.apache.atlas:impala-bridge:3.0.0-SNAPSHOT 
> (/home/lina/sw/atlas/addons/impala-bridge/pom.xml) has 1 error
> [ERROR] 'dependencies.dependency.version' for 
> org.apache.atlas:impala-hook-api:jar is missing. @ 
> org.apache.atlas:impala-bridge:[unknown-version], 
> /home/lina/sw/atlas/addons/impala-bridge/pom.xml, line 56, column 17
> [ERROR]   
> [ERROR]   The project org.apache.atlas:impala-bridge-shim:3.0.0-SNAPSHOT 
> (/home/lina/sw/atlas/addons/impala-bridge-shim/pom.xml) has 1 error
> [ERROR] 'dependencies.dependency.version' for 
> org.apache.atlas:impala-hook-api:jar is missing. @ 
> org.apache.atlas:impala-bridge-shim:[unknown-version], 
> /home/lina/sw/atlas/addons/impala-bridge-shim/pom.xml, line 38, column 17
> 
> Sridhar K wrote:
> You should put the dependency root pom.xml file. Please check  root>/pom.xml for samples..here is a sample from root pom.xml file. Make sure 
> that you are adding in  dependencyManagement section..
> 
> 
> org.apache.atlas
> atlas-plugin-classloader
> ${project.version}
> 

done


- Na


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/#review215416
---


On May 21, 2019, 10:10 p.m., Na Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70619/
> ---
> 
> (Updated May 21, 2019, 10:10 p.m.)
> 
> 
> Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Integrate with Impala hook using Impala private jar with post exec hook
> 
> changes:
> 
> 1) handle changed lineage record format change
> The lineage record format has changed from having seperate table vertex 
> 
>   {  
>  "id":4,
>  "vertexType":"TABLE",
>  "vertexId":"db_1.table_1",
>  "createTime":155475
>   }
>   
> To embed the table create time inside column vertex
> 
> {
>   "id": 3,
>   "vertexType": "COLUMN",
>   "vertexId": "db_1.table_1.tinyint_col",
>   "metadata": {
> "createTime": 1557860978,
> "tableName": "db_1.table_1"
>   }
> }
> 
> 2) add Impala hook API code to allow Atlas integration code to compile 
> without Impala updates
> 
> 
> Diffs
> -
> 
>   addons/impala-bridge-shim/pom.xml PRE-CREATION 
>   
> addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
>  PRE-CREATION 
>   addons/impala-bridge/pom.xml 8e7e952 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
>  fdb6748 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
>  afe296c 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
>  e4a38a3 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
>  7c1103a 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
>  82672c9 
>   
> addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
>  PRE-CREATION 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
>  cc62955 
>   
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
>  6e4d321 
>   addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
>   addons/impala-bridge/src/test/resources/impala4.json PRE-CREATION 
>   addons/impala-bridge/src/test/resources/impala5.json 854969b 
>   addons/impala-bridge/src/test/resources/impala6.json f136180 
>   addons/impala-bridge/src/test/resources/impala7.json f9ee670 
>   addons/impala-hook-api/pom.xml PRE-CREATION 
>   
> addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
>  PRE-CREATION 
>   
>

Re: Review Request 70619: ATLAS-3080: Integration with Impala Hook API

2019-05-21 Thread Na Li via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70619/
---

(Updated May 21, 2019, 10:10 p.m.)


Review request for atlas, Fredy Wijaya, Madhan Neethiraj, and Sarath 
Subramanian.


Repository: atlas


Description
---

Integrate with Impala hook using Impala private jar with post exec hook

changes:

1) handle changed lineage record format change
The lineage record format has changed from having seperate table vertex 

  {  
 "id":4,
 "vertexType":"TABLE",
 "vertexId":"db_1.table_1",
 "createTime":155475
  }
  
To embed the table create time inside column vertex

{
  "id": 3,
  "vertexType": "COLUMN",
  "vertexId": "db_1.table_1.tinyint_col",
  "metadata": {
"createTime": 1557860978,
"tableName": "db_1.table_1"
  }
}

2) add Impala hook API code to allow Atlas integration code to compile without 
Impala updates


Diffs
-

  addons/impala-bridge-shim/pom.xml PRE-CREATION 
  
addons/impala-bridge-shim/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
 PRE-CREATION 
  addons/impala-bridge/pom.xml 8e7e952 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaHook.java 
PRE-CREATION 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/ImpalaLineageHook.java
 fdb6748 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/BaseImpalaEvent.java
 afe296c 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/hook/events/CreateImpalaProcess.java
 e4a38a3 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/IImpalaLineageHook.java
 7c1103a 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertex.java
 82672c9 
  
addons/impala-bridge/src/main/java/org.apache.atlas.impala/model/LineageVertexMetadata.java
 PRE-CREATION 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java
 cc62955 
  
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java
 6e4d321 
  addons/impala-bridge/src/test/resources/impala3.json 6a7d171 
  addons/impala-bridge/src/test/resources/impala4.json PRE-CREATION 
  addons/impala-bridge/src/test/resources/impala5.json 854969b 
  addons/impala-bridge/src/test/resources/impala6.json f136180 
  addons/impala-bridge/src/test/resources/impala7.json f9ee670 
  addons/impala-hook-api/pom.xml PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java
 PRE-CREATION 
  
addons/impala-hook-api/src/main/java/org/apache/impala/hooks/QueryExecHook.java 
PRE-CREATION 
  pom.xml 98c2d69 


Diff: https://reviews.apache.org/r/70619/diff/8/


Testing
---

add integration tests 
ImpalaLineageToolIT.testCreateViewFromFileWithoutCreateTime to handle the 
situation that there is no createTime in lineage record
change the file content for other integration tests in ImpalaLineageToolIT to 
handle new lineage record format


Thanks,

Na Li