Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

2016-05-12 Thread Suma Shivaprasad 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/
---

(Updated May 12, 2016, 7:01 a.m.)


Review request for atlas.


Changes
---

Fixed lineage for hdfs paths


Bugs: ATLAS-619
https://issues.apache.org/jira/browse/ATLAS-619


Repository: atlas


Description
---

Normlized all literals in a query and replacing them CONSTANT literals. Storing 
recently run queries in  a new attribute for hive_process - recentQueries


Diffs (updated)
-

  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 
8fced05 
  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
 2494675 
  
addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
 4e2a06f 
  addons/hive-bridge/pom.xml eeb2aa4 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
ffd5d22 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 6418770 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java 
PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
 PRE-CREATION 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
 PRE-CREATION 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
da21195 
  addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
ab7e6ee 
  
addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java 
2820169 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 267e228 
  
addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
 de67c39 
  
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 a017536 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
9f69940 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 66e1365 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala
 73b3526 
  webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
  webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
  
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
 bc02f90 

Diff: https://reviews.apache.org/r/47107/diff/


Testing
---

Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT


Thanks,

Suma Shivaprasad

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

2016-05-12 Thread Shwetha GS 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/#review132845
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
468)


rename to normaliseQueryStr() and pass just the query str



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
475)


change to warn. Change to 'catch Exception' to handle RuntimeException?



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
601)


rename queryStr to normalisedQueryStr



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
622)


Can you file a bug to handle hive_process update - to update recentQueries



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
631)


rename to addIOTableName() or addDatasetName()?



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
640)


Why add input/output table name? Its part of the normalised query already, 
right? 

Sorry, if we have discussed this before, I can't recollect



addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 
641)


Add a delimiter like :?



addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 (line 83)


not required. Use HiveDataTypes.HIVE_TABLE.getName()



addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
 (line 56)


can we always throw exception from here. anyway, its ignored in doRewrites()



addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
 (line 51)


typo - quuery



addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
 (line 64)


Isn't it possible to re-write this to expColumn = NUMBER_LITERAL?



addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
 (line 67)


remove catch



addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
(line 109)


extract getSqoopProcessName(data, clusterName) to a variable and re-use



repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 (line 555)


no changes, revert this file?



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 192)


What about for dataset - don't you want to do the same changes for dataset 
as well? or planning to do it in another jira?



repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java (line 330)


you can use name for qualified name as well. So, don't need an extra 
argument?



typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala
 (line 246)


revert file?


Normalising query literals is nice. Thanks. Few things:
1. Do we want to skip normalisation for hive_process created as part of create 
table - as there won't be hive_process update in this case and actual query is 
more useful in this case
2. I don't know enough of hive that I could review AST re-write. Can you check 
with hive team to review that part?

- Shwetha GS


On May 12, 2016, 7:01 a.m., Suma Shivaprasad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47107/
> ---
> 
> (Updated May 12, 2016, 7:01 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-619
> https://issues.apache.org/jira/browse/ATLAS-619
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Normlized all literals in a query and replacing them CONSTANT literals. 
> Storing recently run queries in  a new attribute for hive_process - 
> recentQueries
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8fced05 
>   
> addons/falcon-bridge/src

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

2016-05-12 Thread Suma Shivaprasad 


> On May 12, 2016, 9:21 a.m., Shwetha GS wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java, 
> > line 744
> > 
> >
> > Why add input/output table name? Its part of the normalised query 
> > already, right? 
> > 
> > Sorry, if we have discussed this before, I can't recollect

Its useful mostly in the case of HDFS paths, temporary tables . In case of HDFS 
paths, the whole path is replaces with StringLiteral . So if a user does export 
table to 'hdfs path' twice with two different paths. However , it also means 
that partition based paths will also be created which we wanted to avoid . But 
I dont see an easy way around this since we are not able to detect partition 
dirs inside a path easily. But its better than not showing lineage. Here I have 
added in all cases instead of only in specific cases.


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/#review132845
---


On May 12, 2016, 7:01 a.m., Suma Shivaprasad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47107/
> ---
> 
> (Updated May 12, 2016, 7:01 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-619
> https://issues.apache.org/jira/browse/ATLAS-619
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Normlized all literals in a query and replacing them CONSTANT literals. 
> Storing recently run queries in  a new attribute for hive_process - 
> recentQueries
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8fced05 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
>  2494675 
>   
> addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
>  4e2a06f 
>   addons/hive-bridge/pom.xml eeb2aa4 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffd5d22 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  6418770 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> da21195 
>   addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> ab7e6ee 
>   
> addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java
>  2820169 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  267e228 
>   
> addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
>  de67c39 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a017536 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  9f69940 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> 66e1365 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala
>  73b3526 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
>   webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
>  bc02f90 
> 
> Diff: https://reviews.apache.org/r/47107/diff/
> 
> 
> Testing
> ---
> 
> Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

2016-05-12 Thread Suma Shivaprasad 


> On May 12, 2016, 9:21 a.m., Shwetha GS wrote:
> > addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java,
> >  line 64
> > 
> >
> > Isn't it possible to re-write this to expColumn = NUMBER_LITERAL?

since its cast(string) . Its not possible to since hive parser interprets that 
as string literal.


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/#review132845
---


On May 12, 2016, 7:01 a.m., Suma Shivaprasad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47107/
> ---
> 
> (Updated May 12, 2016, 7:01 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-619
> https://issues.apache.org/jira/browse/ATLAS-619
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Normlized all literals in a query and replacing them CONSTANT literals. 
> Storing recently run queries in  a new attribute for hive_process - 
> recentQueries
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8fced05 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
>  2494675 
>   
> addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
>  4e2a06f 
>   addons/hive-bridge/pom.xml eeb2aa4 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffd5d22 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  6418770 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> da21195 
>   addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> ab7e6ee 
>   
> addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java
>  2820169 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  267e228 
>   
> addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
>  de67c39 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a017536 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  9f69940 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> 66e1365 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala
>  73b3526 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
>   webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
>  bc02f90 
> 
> Diff: https://reviews.apache.org/r/47107/diff/
> 
> 
> Testing
> ---
> 
> Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

2016-05-12 Thread Suma Shivaprasad 


> On May 12, 2016, 9:21 a.m., Shwetha GS wrote:
> > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java,
> >  line 192
> > 
> >
> > What about for dataset - don't you want to do the same changes for 
> > dataset as well? or planning to do it in another jira?

this is being tracked in ATLAS-542. Will take care as part of that. It may not 
be needed if we go for qualifiedName as expressions.


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/#review132845
---


On May 12, 2016, 7:01 a.m., Suma Shivaprasad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47107/
> ---
> 
> (Updated May 12, 2016, 7:01 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-619
> https://issues.apache.org/jira/browse/ATLAS-619
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Normlized all literals in a query and replacing them CONSTANT literals. 
> Storing recently run queries in  a new attribute for hive_process - 
> recentQueries
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8fced05 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
>  2494675 
>   
> addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
>  4e2a06f 
>   addons/hive-bridge/pom.xml eeb2aa4 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffd5d22 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  6418770 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> da21195 
>   addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> ab7e6ee 
>   
> addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java
>  2820169 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  267e228 
>   
> addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
>  de67c39 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a017536 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  9f69940 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> 66e1365 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/InstanceSerialization.scala
>  73b3526 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
>   webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
>  bc02f90 
> 
> Diff: https://reviews.apache.org/r/47107/diff/
> 
> 
> Testing
> ---
> 
> Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

2016-05-13 Thread Suma Shivaprasad 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/
---

(Updated May 13, 2016, 7:11 a.m.)


Review request for atlas.


Changes
---

Fixed review comments


Bugs: ATLAS-619
https://issues.apache.org/jira/browse/ATLAS-619


Repository: atlas


Description
---

Normlized all literals in a query and replacing them CONSTANT literals. Storing 
recently run queries in  a new attribute for hive_process - recentQueries


Diffs (updated)
-

  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 
8fced05 
  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
 2494675 
  
addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
 4e2a06f 
  addons/hive-bridge/pom.xml eeb2aa4 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
ffd5d22 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 6418770 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java 
PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
 PRE-CREATION 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
 PRE-CREATION 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
da21195 
  addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
ab7e6ee 
  
addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java 
2820169 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 267e228 
  
addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
 de67c39 
  
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 a017536 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
9f69940 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 66e1365 
  webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
  webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
  
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
 bc02f90 

Diff: https://reviews.apache.org/r/47107/diff/


Testing
---

Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT


Thanks,

Suma Shivaprasad

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

2016-05-13 Thread Suma Shivaprasad 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/
---

(Updated May 13, 2016, 7:20 a.m.)


Review request for atlas.


Bugs: ATLAS-619
https://issues.apache.org/jira/browse/ATLAS-619


Repository: atlas


Description
---

Normlized all literals in a query and replacing them CONSTANT literals. Storing 
recently run queries in  a new attribute for hive_process - recentQueries


Diffs (updated)
-

  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 
8fced05 
  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
 2494675 
  
addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
 4e2a06f 
  addons/hive-bridge/pom.xml eeb2aa4 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
ffd5d22 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 6418770 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java 
PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
 PRE-CREATION 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
 PRE-CREATION 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
da21195 
  addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
ab7e6ee 
  
addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java 
2820169 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 267e228 
  
addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
 de67c39 
  
repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 a017536 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
9f69940 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 66e1365 
  webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
  webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
  
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
 bc02f90 

Diff: https://reviews.apache.org/r/47107/diff/


Testing
---

Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT


Thanks,

Suma Shivaprasad

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

2016-05-13 Thread Suma Shivaprasad 


> On May 12, 2016, 9:21 a.m., Shwetha GS wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java, 
> > line 726
> > 
> >
> > Can you file a bug to handle hive_process update - to update 
> > recentQueries

ATLAS-782


- Suma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/#review132845
---


On May 13, 2016, 7:20 a.m., Suma Shivaprasad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47107/
> ---
> 
> (Updated May 13, 2016, 7:20 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-619
> https://issues.apache.org/jira/browse/ATLAS-619
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Normlized all literals in a query and replacing them CONSTANT literals. 
> Storing recently run queries in  a new attribute for hive_process - 
> recentQueries
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8fced05 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
>  2494675 
>   
> addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
>  4e2a06f 
>   addons/hive-bridge/pom.xml eeb2aa4 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffd5d22 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  6418770 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> da21195 
>   addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> ab7e6ee 
>   
> addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java
>  2820169 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  267e228 
>   
> addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
>  de67c39 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a017536 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  9f69940 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> 66e1365 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
>   webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
>  bc02f90 
> 
> Diff: https://reviews.apache.org/r/47107/diff/
> 
> 
> Testing
> ---
> 
> Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

2016-05-13 Thread Shwetha GS 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/#review133067
---


Fix it, then Ship it!





addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
(line 110)


change to - procRef.set(AtlasClient.REFERENCEABLE_ATTRIBUTE_NAME, 
sqoopProcessName)



repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
 (line 560)


no changes, revert file


- Shwetha GS


On May 13, 2016, 7:20 a.m., Suma Shivaprasad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47107/
> ---
> 
> (Updated May 13, 2016, 7:20 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-619
> https://issues.apache.org/jira/browse/ATLAS-619
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Normlized all literals in a query and replacing them CONSTANT literals. 
> Storing recently run queries in  a new attribute for hive_process - 
> recentQueries
> 
> 
> Diffs
> -
> 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java
>  8fced05 
>   
> addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
>  2494675 
>   
> addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
>  4e2a06f 
>   addons/hive-bridge/pom.xml eeb2aa4 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> ffd5d22 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
>  6418770 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
> da21195 
>   addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
>   
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
> ab7e6ee 
>   
> addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java
>  2820169 
>   
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
>  267e228 
>   
> addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
>  de67c39 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java
>  a017536 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  9f69940 
>   repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 
> 66e1365 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
>   webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
>  bc02f90 
> 
> Diff: https://reviews.apache.org/r/47107/diff/
> 
> 
> Testing
> ---
> 
> Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>

Re: Review Request 47107: ATLAS_619 Normalize literals for hive queries

2016-05-13 Thread Suma Shivaprasad 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47107/
---

(Updated May 13, 2016, 3:37 p.m.)


Review request for atlas.


Changes
---

fixed comments


Bugs: ATLAS-619
https://issues.apache.org/jira/browse/ATLAS-619


Repository: atlas


Description
---

Normlized all literals in a query and replacing them CONSTANT literals. Storing 
recently run queries in  a new attribute for hive_process - recentQueries


Diffs (updated)
-

  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/hook/FalconHook.java 
8fced05 
  
addons/falcon-bridge/src/main/java/org/apache/atlas/falcon/model/FalconDataModelGenerator.java
 2494675 
  
addons/falcon-bridge/src/test/java/org/apache/atlas/falcon/hook/FalconHookIT.java
 4e2a06f 
  addons/hive-bridge/pom.xml eeb2aa4 
  addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
ffd5d22 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java
 6418770 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/ASTRewriter.java 
PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/HiveASTRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/LiteralRewriter.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteContext.java
 PRE-CREATION 
  
addons/hive-bridge/src/main/java/org/apache/atlas/hive/rewrite/RewriteException.java
 PRE-CREATION 
  
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveLiteralRewriterTest.java
 PRE-CREATION 
  addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 
da21195 
  addons/hive-bridge/src/test/resources/hive-site.xml f1facb8 
  addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 
ab7e6ee 
  
addons/sqoop-bridge/src/test/java/org/apache/atlas/sqoop/hook/SqoopHookIT.java 
2820169 
  
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java
 267e228 
  
addons/storm-bridge/src/main/scala/org/apache/atlas/storm/model/StormDataModel.scala
 de67c39 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
9f69940 
  repository/src/test/java/org/apache/atlas/BaseHiveRepositoryTest.java 66e1365 
  webapp/src/main/java/org/apache/atlas/examples/QuickStart.java 70dce6b 
  webapp/src/test/java/org/apache/atlas/examples/QuickStartIT.java cdf6049 
  
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java
 bc02f90 

Diff: https://reviews.apache.org/r/47107/diff/


Testing
---

Added UT - HiveAStREwriteTest and IT asserttions in HiveHookIT


Thanks,

Suma Shivaprasad